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CRESST  HUMAN  PERFORMANCE  KNOWLEDGE  MAPPING  SYSTEM 


Gregory  K.  W.  K.  Chung,  Joanne  K.  Michiuye,  David  G.  Brill,  Ravi  Sinha, 
Farzad  Saadat,  Linda  F.  de  Vries,  Girlie  C.  Delacruz,  William  L.  Bewley, 

and  Eva  L.  Baker 

CRESST/University  of  California,  Los  Angeles 


Abstract 

In  this  study  we  report  on  the  design  and  development  of  a  knowledge 
mapping  tool  intended  to  measure  content  understanding.  We  review  knowledge 
mapping  scoring  methods  and  current  online  mapping  systems.  We  then  describe 
the  overall  design  approach,  functionality,  scoring,  usability  testing,  and  authoring 
capabilities  of  our  system.  While  there  exist  several  tools  available  to  construct 
knowledge  maps,  CRESST's  knowledge  mapping  tool  is  one  of  the  only  systems 
designed  specifically  for  assessment  purposes,  the  only  system  that  can  support 
multiple  assessment  formats,  and  the  only  system  with  an  empirical  base. 
Limitations  of  the  system  and  possible  next  steps  are  discussed. 
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Introduction 

The  Navy-wide  distributed  learning  training  vision  is  to  deliver  quality 
training — to  the  right  people,  at  the  right  time,  and  at  the  right  place — as  part  of  a 
career-long  training  continuum  supporting  Navy  operational  readiness  and 
personal  excellence.  Achieving  this  vision  of  delivering  learner-centric  training  to 
individuals  will  require  maximizing  the  overlap  between  instruction — its  format, 
pacing,  and  complexity — and  the  individual's  cognitive,  affective,  and  learning 
predispositions  and  preferences,  deficiencies,  and  strengths.  Attaining  this 
capability  will  help  equip  Sailors  and  Marines  with  effective  mission-essential 
competencies  when  and  where  needed  at  an  affordable  cost. 

A  critical  first  step  in  developing  learner-centric  systems  is  gathering  quality 
information  about  an  individual's  competency  in  a  skill  or  knowledge  domain.  Such 
information  includes,  for  example,  an  estimate  of  what  trainees  know  prior  to 
training,  how  much  they  have  learned  from  training,  how  well  they  may  perform  in 
a  future  target  situation,  or  whether  to  recommend  remediation  content  to  bolster  a 
trainees'  knowledge. 

The  focus  of  this  report  is  on  the  development  of  a  tool  designed  to  assess  a 
trainee's  understanding  of  a  content  domain  via  graphical  representation.  The  tool, 
referred  to  as  the  CRESST  Human  Performance  Knowledge  Mapping  Tool 
(HPKMT),  requires  trainees  to  express  their  understanding  of  a  content  area  by 
creating  knowledge  maps.  Knowledge  maps  are  network  representations,  where 
nodes  represents  concepts  and  links  represent  the  relationship  between  two 
concepts.  The  questions  that  guided  the  development  of  the  HPKMT  were: 

•  What  are  the  existing  scoring  methods  for  knowledge  maps?  While  we  had 
already  had  an  approach  for  scoring  knowledge  maps,  we  were  interested 
in  incorporating  any  new  developments  in  scoring  approaches. 

•  What  are  the  existing  software  packages  that  can  be  used  for  assessment 
purposes?  At  a  minimum,  we  required  the  package  to  support  the 
knowledge  mapping  format.  In  addition,  we  were  interested  in  a  tool  that 
could  support  other,  unforeseen  formats.  In  a  distributed  learning  context,  it 
would  be  extremely  useful  to  have  a  common  user  interface  with  a  single 
metaphor  to  deliver  different  assessment  formats.  Thus,  we  reviewed 
existing  software  to  determine  whether  there  existed  commercial,  off-the- 
shelf  (COTS)  products  that  could  be  used. 
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•  What  are  the  key  functional  requirements  for  such  an  assessment  tool?  In  a 
preview  of  our  findings,  we  concluded  from  our  evaluation  of  COTS 
products  that  there  existed  no  product  that  would  satisfy  our  requirements 
for  an  assessment  tool. 

There  are  three  main  sections  of  this  report.  In  the  first  section,  we  situate  the 
HPKMT  within  the  general  area  of  knowledge  mapping  for  assessment  purposes. 
We  first  review  the  different  scoring  approaches  that  have  been  used  to  evaluate  the 
quality  of  knowledge  maps.  Our  focus  is  on  scoring  because  this  aspect  critically 
impacts  the  validity  of  the  inferences  drawn  about  trainee  performance.  We  then 
discuss  briefly  current  online  mapping  systems  and  our  judgment  of  their  ease  of 
use  and  scoring  capability.  The  second  major  section  discusses  the  HPKMT 
functionality.  Using  the  review  of  scoring,  the  review  of  online  products,  and  our 
prior  work  with  online  knowledge  mapping,  we  describe  our  design.  We  report  on 
our  experience  with  using  the  HPKMT  in  both  a  usability  test  and  field  test.  The  last 
major  section  is  a  description  of  the  authoring  prototype.  We  provide  a  functional 
description  with  screen  shots  for  illustration  purposes.  We  then  discuss  limitations 
of  this  work  and  identify  next  steps. 

Overview  of  Scoring  Methods  for  Knowledge  Maps 

A  presumed  critical  capability  of  an  assessment  in  a  distributed  learning  setting 
is  automated  scoring.  A  critical  validity  issue  of  an  assessment  is  the  scoring, 
regardless  of  automated  capability.  In  this  section  we  briefly  describe  the  different 
types  of  scoring  and  provide  examples  of  their  use.  For  in-depth  reviews  of 
assessment  issues  related  to  knowledge  maps,  see  Ruiz-Primo  and  Shavelson  (1996). 

In  general,  scoring  knowledge  maps  can  be  referent-based  or  referent-free. 
Referent-based  methods  compare  a  student's  map  against  a  referent  map  (e.g.,  an 
expert's  map  or  other  gold  standard).  Referent-free  methods  evaluate  the  student's 
map  against  a  rubric  or  with  other  criteria  (e.g.,  judging  the  quality  of  the 
propositions  [node-link-node  tuple],  or  counting  the  number  of  concepts  in  the 
map).  In  either  case,  different  scoring  approaches  use  to  different  degrees  the 
configural  and  semantic  properties  of  the  network.  Table  1  summarizes  scoring 
methods. 
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Table  1 

Simplified  Summary  of  Knowledge  Mapping  Scoring  Methods 

Configural 

Semantic 

Referent-free  Explicitly  scores  a  map  or 
elements  of  a  map  on  its 
structural  aspect  (e.g., 
considering  degree  of 
hierarchical  organization). 


Explicitly  scores  a  map  or  elements  of  a  map 
on  its  semantic  aspect  (e.g.,  scoring  quality  of 
propositions). 

Example  applications: 


Example  application: 
Novak  and  Gowin  (1984). 


Osmundson,  Chung,  Herl,  and  Klein  (1999). 

Shavelson  (Ruiz-Primo,  Schultz,  Li,  & 
Shavelson,  2001) 


Referent-based  Compares  the  network 

structure  of  a  student's  map 
and  the  referent  map.  Does  not 
take  into  account  the  meaning 
of  the  relationships. 

Example  application: 

Herl,  Baker,  and  Niemi  (1996). 


Compares  the  semantic  structure  of  a  student's 
map  and  the  referent  map  (e.g.,  proposition- 
by-proposition  comparison  between  a 
student's  map  and  an  expert's  map).  Ignores 
the  configural  aspects  of  the  network. 

Example  applications: 

Herl  et  al.  (1996). 

Osmundson  et  al.  (1999). 


Referent-Free  Methods 

The  scoring  procedure  specified  by  Novak  and  Gowin  (1984)  is  one  of  the 
earliest  and  most  commonly  used  methods  of  scoring  knowledge  maps.  Their 
method  considers  hierarchy  as  an  important  component  of  the  scoring,  as  well  as 
propositions,  cross-links,  and  examples.  In  terms  of  hierarchy,  credit  is  given  for 
each  hierarchical  level  showing  subordinate  concepts  at  a  lower  level  as  more 
specific  than  their  parent  concepts.  Each  valid  and  meaningful  proposition  is  also 
credited,  as  are  examples  and  cross-links.  Cross-links  are  links  between  different 
hierarchical  levels.  Novak  and  Gowin's  scoring  scheme  is  weighted  heavily  towards 
the  hierarchical  structure  of  the  map.  The  theoretical  rationale  for  this  scoring 
scheme  is  AusabeTs  theory  of  learning,  particularly  the  idea  of  subsumption  (new 
ideas  can  be  subsumed  under  more  general  concepts)  and  progressive 
differentiation  (as  learning  occurs,  there  is  more  differentiation  among  the  concepts, 
which  is  shown  by  the  inclusion  of  more  propositions  and  cross-links).  Appendix  A 
contains  Novak  and  Gowin's  scoring  procedure. 
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Evidence  from  several  studies  suggests  that  Novak  and  Gowin's  (1984)  scoring 
scheme  can  differentiate  between  high-  and  low-knowledge  students  in  biology 
(Markham,  Mintzes,  &  Jones,  1994)  and  between  first-year  and  advanced  pediatric 
residents  studying  seizures  (West,  Pomeroy,  Park,  Gerstenberger,  &  Sandoval,  2000). 
This  scoring  scheme  also  appears  to  be  sensitive  to  learning,  as  student  map  scores 
increased  over  instruction  (Pearsall,  Skipper,  &  Mintzes,  1997;  West  et  al.,  2000). 

A  second  scoring  scheme  that  is  commonly  used  considers  only  the 
propositions  contained  in  the  map  and  not  the  configural  aspects.  This  method  is  to 
rate  the  quality  of  the  propositions  in  the  map.  Each  proposition  is  evaluated  in 
terms  of  its  accuracy.  For  example,  Ruiz-Primo  and  colleagues  used  a  proposition 
accuracy  score  as  one  measure  of  the  quality  of  students'  knowledge  maps  (Ruiz- 
Primo,  Schultz,  Li,  &  Shavelson,  1997a,  1997b;  Ruiz-Primo  et  al.,  2001).  Each 
proposition  in  a  students'  map  was  scored  on  a  5-point  scale,  ranging  from  0 
(invalid /inaccurate)  to  4  (complete  and  correct  and  showing  a  deep  understanding 
of  the  relation  between  two  concepts).  Ruiz-Primo  and  colleagues  found  that 
students'  proposition  accuracy  scores  differentiated  high-knowledge  students  from 
low-knowledge  students  (e.g.,  Ruiz-Primo  et  al.,  1997a,  1997b)  and  students'  map 
scores  were  moderately  correlated  ( r  between  .40  to  .50)  with  other  measures  of 
content  knowledge  formats  (e.g.,  essays,  multiple  choice  tests).  Similar  relationships 
have  been  found  between  knowledge  map  proposition  accuracy  scores  and 
classroom  end-of-unit  tests  and  standardized  tests  of  reading,  math,  and  science 
(Rice,  Ryan,  &  Samson,  1998),  and  between  knowledge  maps  and  physics  problem 
solving  (Austin  &  Shore,  1995). 

Referent-Based  Methods 

Referent-based  methods  compare  a  student's  map  against  a  criterion  map. 
Example  referents  include  an  expert's  map,  a  composite  map  of  experts,  or  the 
instructor's  map.  The  essential  measure  is  the  number  of  propositions  in  the  student 
map  that  are  also  in  the  referent  map.  Several  studies  have  investigated  the  technical 
properties  of  this  approach.  For  example,  Ruiz-Primo  et  al.  (2001),  in  addition  to 
using  proposition  accuracy  scores,  also  scored  students'  maps  against  an  expert's 
map.  The  correlation  between  the  proposition  accuracy  score  and  expert-based  score 
was  sufficiently  high  for  Ruiz-Primo  et  al.  to  conclude  that  an  expert-based  method 
was  the  most  efficient  scoring  method  (i.e.,  in  terms  of  scoring  time  and  reliability  of 
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scores).  Similar  results  were  found  by  Osmundson  et  al.  (1999)  and  Chung,  Harmon, 
and  Baker  (2001). 

The  findings  of  Ruiz-Primo  et  al.  (2001)  are  consistent  with  earlier  work  by  Herl 
(1995),  Herl  et  al.  (1996),  and  Osmundson  et  al.  (1999).  In  general,  scoring  student 
knowledge  maps  using  expert-based  referents  has  been  found  to  discriminate 
between  experts  and  novices  (Herl,  1995;  Herl  et  al.,  1996),  discriminate  between 
different  levels  of  student  performance  (Herl,  1995;  Herl  et  al.,  1996),  relate 
moderately  to  external  measures  (Aguirre-Muoz,  2000;  Herl,  1995;  Herl  et  al.,  1996; 
Klein,  Chung,  Osmundson,  Herl,  &  O'Neil,  2002;  Lee,  2000;  Osmundson  et  al.,  1999), 
detect  changes  in  learning  (Chung  et  al.,  2001;  Osmundson  et  al.,  1999;  Schacter, 
Herl,  Chung,  Dennis,  &  O'Neil,  1999),  and  be  sensitive  to  language  proficiency 
(Aguir re-Munoz,  2000;  Lee,  2000). 

The  final  type  of  scoring  is  to  simply  compare  the  network  topology  of  a 
student's  map  and  the  referent  map.  Herl  et  al.  (1996)  investigated  the  utility  of  this 
approach  and  found  high  correlations  between  scores  based  on  a  comparison  of  the 
network  topology  and  scores  based  on  the  overlap  of  propositions  between  the 
student  and  expert  map. 

Summary 

A  variety  of  methods  have  been  developed  and  researched  for  scoring 
knowledge  maps.  Using  expert-based  referent  maps  as  scoring  templates  appears  to 
be  an  efficient  means  of  scoring  and  produces  results  comparable  to  proposition 
quality  ratings.  This  finding  has  important  implications  for  automated  scoring 
purposes,  as  will  be  discussed  in  the  next  section. 

Review  of  Existing  Mapping  Software 

The  purpose  of  this  review  was  to  identify  COTS  software  that  could  be  used 
for  knowledge  mapping  for  assessment  purposes.  Our  guideline  for  inclusion  was 
practical:  If  we  were  to  adopt  existing  mapping  software  for  our  own  assessment- 
focused  applications,  which  software  would  we  at  least  review  for  consideration? 
We  reviewed  15  packages  that  were  designed  specifically  for  knowledge  mapping  or 
could  be  used  to  construct  knowledge  maps.  Two  raters  judged  the  software  on  the 
following  criteria:  (a)  intuitiveness  of  the  interface,  and  (b)  automated  scoring 
capability.  One  rater  conducted  the  usability  study  described  in  the  next  section,  and 
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the  second  rater  had  no  experience  with  knowledge  mapping  but  had  experience 
using  graphical  packages.  Each  criterion  is  briefly  described. 

Intuitiveness  of  the  Interface 

One  of  the  critical  lessons  learned  from  our  field  experience  using  computer- 
based  assessments  is  that  the  user  interface  needs  to  be  as  easy  to  use  as  possible. 
Typical  field  conditions  include  limited  time  in  a  group  instructional  setting  in  a 
non-computer  lab  room  (e.g.,  classroom,  library,  or  side  room).  The  configuration  of 
the  setting  has  important  training  implications.  Usually  group  instruction  is 
required  on  how  to  use  the  software  and  users  are  not  seated  in  front  of  a  computer 
using  the  software  during  training.  If  the  user  interface  is  complex,  then  questions 
arise  during  the  task.  Under  conditions  where  there  is  ample  time,  questions  are 
usually  not  a  problem;  however,  when  there  are  time  constraints  (e.g.,  a  class  period) 
and  the  purpose  is  assessing  student  knowledge,  then  user-interface  complexities 
may  introduce  undesirable  effects  that  are  unrelated  to  the  task.  In  a  distributed 
learning  setting,  we  expect  that  the  interface  will  play  an  even  more  important  role 
as  there  will  be  no  face-to-face  support.  Thus,  ease  of  use  is  an  important  criterion 
for  practical  and  validity  reasons. 

Scoring  Capability 

A  second  criterion  is  that  if  the  intention  is  to  use  knowledge  maps  for 
assessment  purposes  in  a  distributed  learning  context,  then  as  a  practical  matter 
there  needs  to  be  automated  scoring  capability.  Further,  it  is  not  enough  to  be  able  to 
"score"  a  map.  There  should  be  validity  evidence  associated  with  the  scoring 
algorithm,  either  by  implementing  one  of  the  approaches  discussed  in  the  previous 
section,  or  by  documenting  the  relationship  between  the  scoring  algorithm  and  other 
external  measures,  theory,  or  the  extant  literature. 
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Table  2 

Summary  of  Product  Evaluations _ 

Intuitiveness  of 
the  user 

interface  Automated 

-  scoring 


Product 

R1 

R2 

capability 

Comments 

IHMC  Concept  Map 
Software 

L 

L 

No 

Too  hard  to  use  without  over¬ 
relying  on  Help  function. 

Inspiration 

H 

H 

No 

Easy  to  use,  impressive  range  of 
choices. 

Mind  Matrix 

H 

H 

Yes 

Easy  to  use. 

LifeMap 

L 

M 

No 

Hard  to  use.  Icons  are  not  clear. 

Thinking  Maps 

M 

H 

No 

Cannot  add  relation  labels. 

Concept  Draw  MindMap 

M 

M 

No 

Interface  is  intimidating.  Difficult 
to  link  subcategories. 

Semantica 

Evaluation  copy 

unavailable 

Visual  Mind 

M 

H 

No 

Cannot  add  relation  labels. 

Smart  Ideas 

H 

H 

No 

Easy  to  use.  Linking  in  particular 
is  very  clear. 

Mind  Mapper 

M 

H 

No 

Difficult  to  find  how  to  add 
relation  labels.  Icons  are 
confusing. 

Fuzzy  Thought  Amplifier 

Evaluation  copy 

unavailable 

Axon  Idea  Processor 

M 

M 

No 

Difficult  to  find  how  to  add 
relation  labels. 

Knowledge  Manager 

Evaluation  copy 

unavailable 

Visio 

H 

H 

No 

Easy  to  use  and  visually 
appealing. 

Decision  Explorer 

L 

L 

No 

Too  hard  to  use. 

Mind  Manager 

M 

M 

No 

Not  easy  to  use. 

VisiMap 

L 

L 

No 

Too  hard  to  use.  Difficult  to  create 
new  concepts. 

SmartDraw 

H 

H 

No 

Easy  to  use  but  it  has  too  many 
features  that  are  not  related  to 
making  concept  maps. 

Note.  H  =  high,  M  =  medium,  L  =  low.  An  annotated  list  is  given  in  Appendix  B. 
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Summary 

The  majority  of  software  that  provide  knowledge  mapping  capability  appear  to 
be  focused  primarily  on  instructional  applications,  including  representing  and 
organizing  information,  and  planning.  Aside  from  our  original  mapping  software 
(Version  1.0,  Chung  &  Baker,  1997),  Mind  Matrix  is  the  only  other  product  that  has 
focused  on  assessment  uses  and  the  only  other  product  that  has  automated  scoring 
capability.  However,  the  product  appears  to  only  support  knowledge  mapping 
representations.  Our  requirement  was  for  a  tool  to  support  knowledge  mapping  at  a 
minimum,  with  the  capability  to  expand  into  other  assessment  formats  as  the  need 
arose. 

In  general,  products  not  specifically  designed  for  knowledge  mapping  tend  to 
be  sophisticated  drawing  packages.  For  products  that  have  been  designed 
specifically  for  knowledge  mapping,  there  is  a  range  of  support  for  different  kinds  of 
maps  and  media.  With  the  exception  of  the  CRESST  system,  there  appears  to  be  little 
empirical  work  directly  associated  with  the  use  of  these  systems. 

From  the  review  of  scoring  algorithms  and  the  review  of  COTS  mapping 
packages,  it  was  clear  that  there  was  no  existing  product  that  was  sufficiently 
compelling  to  adopt  out  of  the  box  for  assessment  purposes.  Thus,  we  decided  to 
design  and  develop  our  own  system.  We  drew  on  our  prior  experience,  the  software 
review,  and  lessons  learned  from  our  Version  1.0  system  to  develop  functional 
requirements. 


HPKMT  Functional  Overview 

The  primary  purpose  of  the  Human  Performance  Knowledge  Mapping  Tool 
(HPKMT)  is  to  support  the  assessment  of  an  individual's  content  understanding  of  a 
particular  domain  in  a  distributed  learning  environment.  There  were  four  design 
goals:  (a)  develop  a  research-based,  empirically  supported  measurement  tool;  (b) 
develop  an  easy-to-use  application  that  can  be  used  by  users  of  different  computer 
experience;  (c)  develop  an  architecture  that  can  support  the  development  of  a 
networking  tool  as  well  as  accommodate  future,  unforeseen  assessment  formats; 
and  (d)  develop  an  architecture  that  can  accommodate  automated  scoring  of  student 
responses  for  all  supported  assessment  formats. 

The  requirements  for  the  HPKMT  were  generated  from  a  variety  of  sources. 
First,  we  reviewed  the  knowledge  mapping  literature  to  get  an  overview  of  how 
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knowledge  maps  were  used  for  assessment  purposes.  We  also  reviewed  existing 
software  from  commercial  and  academic  labs  that  purported  to  provide  the 
capability  to  create  knowledge  maps.  Finally,  we  incorporated  lessons  learned  from 
the  CRESST  research  experience  with  online  knowledge  mapping.  Appendix  C 
contains  an  outline  of  the  functional  requirements  for  the  mapper. 

The  knowledge  mapping  system  reported  herein  is  a  second-generation 
version  of  our  online  mapping  system.  Our  original  version,  reported  in  Chung  and 
Baker  (1997),  was  used  in  numerous  contexts,  from  elementary  school  science 
classrooms  to  adult  learners,  under  a  variety  of  conditions  ranging  from 
instructional  use  to  tests  of  content  knowledge.  The  current  version  incorporates 
lessons  learned  from  this  period.  A  summary  of  lessons  learned  from  this  period  is 
presented  in  Table  3.  Each  issue  is  discussed  further. 
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Table  3 

Lessons  Learned  from  CRESST  Knowledge  Mapping  Studies  (1997-2001) 

Issue 

Version  1.0  capability  and  lesson  learned 

Version  2.0  instantiation 

User 

interface 

Capability:  Simple  interface  that  is  easy  to 
use  and  easy  to  train.  No  special  skills  are 
needed. 

Maintained  simplicity  and  ease  of 
use,  and  adopted  a  modem  interface 
design  for  graphical  drawing. 

Lessons  learned:  Need  to  retain  simple 
interface. 

Created  an  interface  that  is  consistent 
with  familiar  applications  (e.g.. 

Word). 

Task  format 

Capability:  Single  task  format  useful  for 
original  purpose  but  imposes  limits  on  what 
can  be  assessed.  Few  options  to  respond  to 
demands  for  more  and  varied  formats. 

Lessons  learned:  Need  to  develop  a  general 
architecture  that  would  accommodate  the 
development  of  different  assessment 
formats. 

Designed  to  accommodate  multiple 
formats  as  needed.  Provided  the 
capability  for  user-defined  icons  to  be 
uploaded  and  mapped.  Provided 
background  imaging  capability  for 
additional  format  choices. 

Authoring 

capability 

Capability:  Minimal  capability  that  required 
customized  attention. 

Developed  capability  to  more  easily 
stitch  knowledge  mapping  tasks 

Lessons  learned:  Need  to  develop  more 
user-friendly  method  of  authoring 
knowledge  maps. 

together. 

Network 

accessibility 

Capability:  Firewalls  blocked  applets;  thus, 
the  mapper  could  not  be  used  in  numerous 
environments. 

Adopted  Webstart  as  a  way  to  launch 
the  mapper  on  the  client  machine  via 
the  browser. 

Lessons  learned:  Need  to  develop  method 
of  invoking  mapper  that  is  not  sensitive  to 
typical  network  security  constraints. 

Scoring 

Capability:  Scoring  algorithm  successful  but 
operationally  cumbersome. 

Lessons  learned:  Maintain  capability  and 
increase  capability  to  incorporate  multiple 
scoring  approaches. 

Ported  scoring  algorithm  and 
expanded  the  set  of  map  attributes 
that  can  be  scored.  For  example, 
comparisons  are  made  between  a 
student's  map  and  the  criterion  map 
on  the  link  line  type,  color,  and 
weight,  and  the  concept  symbol  type, 
line  type,  color,  and  weight. 

User  Interface 

One  of  the  clearest  successes  we  have  had  with  Version  1.0  is  the  usability  of 
the  software.  Version  1.0  was  simple,  focused,  and  easily  comprehendible  by  a  wide 
range  of  users.  There  were  clearly  some  user-interface  problems,  but  these  problems 
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could  be  handled  via  training.  Once  demonstrated,  most  users  had  little  difficulty 
operating  the  software.  We  have  repeatedly  observed  this  result  across  different 
ages  and  contexts  (e.gv  Chung,  Baker,  &  Cheak,  2002;  Chung  et  al.,  2001;  Chung, 
O'Neil,  &  Herl,  1999;  Herl,  O'Neil,  Chung,  Bianchi  et  al.,  1999;  Herl,  O'Neil,  Chung, 
&  Schacter,  1999;  Klein  et  al.,  2002;  Osmundson  et  al.,  1999;  Schacter  et  al.,  1999). 
Thus,  our  starting  point  for  the  design  of  Version  2.0  was  to  maintain  the  ease  of  use 
and  simplicity  of  the  user  interface. 

In  Version  2.0,  we  adopted  a  modern  interface  for  graphical  drawing 
applications.  As  shown  in  Figure  1,  the  screen  space  was  partitioned  into  two  major 
areas:  (a)  symbol  space,  which  housed  icons;  and  (b)  canvas  space,  which  is  where 
the  drawing  took  place  using  the  symbols  from  the  symbol  space.  System  functions 
common  to  applications  were  provided  as  well  (e.g.,  buttons  to  open  a  file,  save  a 
file,  zoom). 

The  main  goal  for  the  Version  2.0  user  interface  was  for  it  to  be  simple  and 
intuitive.  Such  an  interface  would  translate  into  minimal  training,  explanation,  and 
construct  irrelevant  variance — all  important  operational  features  when  the  tool  is 
used  for  assessment  purposes. 

Basic  Operation 

The  knowledge  mapper  was  designed  to  be  used  with  only  the  mouse. 
Concepts  are  added  to  the  map  by  clicking  and  dragging  the  Concept  icon  onto  the 
canvas.  When  a  concept  is  added  to  the  map,  a  pop-up  menu  of  available  concepts  is 
provided  for  the  user  to  choose  from.  Links  between  concepts  are  made  by  drawing 
a  line  from  one  concept  to  another.  At  the  completion  of  the  link  operation,  a  pop-up 
menu  of  available  links  appears  for  the  user  to  choose  from.  Concepts  and  links  can 
be  changed  by  either  double-clicking  or  right-clicking  on  the  concept  or  link  to  bring 
up  the  label  menu. 
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Figure  1.  User  interface  of  Version  2.0  of  the  CRESST  knowledge  mapper. 


Task  Format 

Version  1.0  was  designed  specifically  for  mapping.  One  goal  for  Version  2.0 
was  to  develop  a  more  general  system  in  terms  of  the  types  of  knowledge  the  system 
can  assess.  Increasingly,  we  found  a  single-representation  format  limiting  (e.g.,  see 
Herl,  O'Neil,  Chung,  Bianchi  et  al.,  1999).  Thus,  for  Version  2.0  we  began  with  the 
assumption  that  assessment  needs  will  evolve  over  time  and  that  a  tool  to  support  a 
variety  of  unforeseen  formats  would  be  desirable. 

The  approach  we  adopted  was  to  provide  the  capability  for  end-users  to  use 
different  icon  sets.  An  icon  set  is  a  collection  of  icons  that  represents  the  elements  of 
the  domain  being  assessed.  For  example,  the  mapper  shown  in  Figure  1  has  an  icon 
set  with  1  icon — a  rectangle.  A  procedural  knowledge  icon  set  might  have  3  icons — 
rectangle  for  process,  diamond  for  decision  making,  and  circle  for  begin/ end. 

The  utility  of  this  approach  is  twofold:  (a)  The  same  codebase  can  be  used  to 
store,  retrieve,  and  display  the  representation.  Changes  occur  at  the  icon-set  level 
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and  not  in  the  software.  A  library  of  icon  sets  can  be  developed  to  represent  different 
assessments  from  the  abstract  (e.g.,  planning)  to  the  concrete  (as  actual  physical 
entities),  (b)  The  assessment  option  is  open-ended.  If  the  domain  can  be  represented 
using  icons,  the  mapper  shell  can  display  it. 

A  second  feature  of  Version  2.0  is  to  provide  the  capability  to  use  different 
background  sets.  Like  icon  sets,  the  mapper  can  have  images  to  display  in  the 
background.  Different  backgrounds  can  be  used  as  insertable  plates,  depending  on 
the  purpose.  For  example,  the  way  we  typically  use  the  mapper  is  with  no 
background  and  the  physical  location  of  the  nodes  has  no  meaning.  But  another 
application  may  make  use  of  the  physical  location.  For  example,  this  feature  was 
used  to  assess  Marines'  knowledge  of  shot  groups  for  a  rifle  marksmanship 
application.  As  shown  in  Figure  2,  the  same  shell  has  been  used  to  administer  a 
"physical"-based  representation.  Physical  placement  of  the  icons  relative  to  other 
icons  and  relative  to  the  background  image  is  used  directly  for  scoring. 
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CRESST  Knowledge  Mapper  -  TASK:  Target  Error 
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Figure  2.  Use  of  background  to  depict  physical-based  layouts. 


Network  Accessibility 

One  restriction  imposed  by  firewalls  is  the  blocking  or  filtering  of  Java  applets 
and  other  objects  (e.g..  Active  X).  Version  1.0  was  applet-based,  and  our  experience 
in  large  school  districts  and  commercial  sites  suggested  that  firewalls  were 
becoming  commonplace  and  highly  restrictive.  Given  the  high  probability  of  the  use 
of  firewalls  by  our  likely  end-users,  we  designed  the  mapper  to  be  client-based.  We 
used  a  technology  called  Webstart,  which  allowed  the  knowledge  mapper  to  be 
launched  as  an  application  on  the  user's  machine  via  the  user's  browser.  This  is  a 
key  feature  because  the  mapper  is  implemented  as  an  application,  not  an  applet 
(thereby  avoiding  firewall  restrictions).  Code  maintenance  is  handled  automatically 
by  Webstart,  which  automatically  checks  for  updates  to  the  mapper  and  downloads 
components  that  have  changed. 
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Automated  Scoring  Capability 

Scoring  of  knowledge  maps.  We  designed  the  scoring  engine  to  be  loosely 
coupled  to  the  system,  thereby  allowing  for  maximal  flexibility  in  the  scoring 
routines.  That  is,  knowledge  mapping  data  are  stored  in  tables  and  databases  that 
are  independently  accessed  by  the  scoring  engine.  This  separation  reduces  the 
complexity  of  the  mapping  code,  increases  maintainability,  and  allows  for  scoring 
algorithms  to  be  developed  independently  of  the  mapper. 

Currently,  we  have  implemented  the  following  scoring  routines  for  the 
knowledge  mapping  representation:  (a)  exact  proposition  matching  and  (b) 
synonym-based  proposition  matching.  The  exact-proposition-matching  algorithm  is 
based  on  the  algorithm  developed  by  Herl  et  al.  (1996).  Exact  proposition  matching 
involves  counting  the  number  of  propositions  in  the  trainee  map  that  also  exist  in 
the  referent  map  (e.g.,  an  expert's  map). 

Synonym  scoring  is  a  more  lenient  method  that  considers  a  match  if  there  exists 
an  intersection  between  the  set  of  synonyms  for  each  term  in  the  trainee  map  and 
each  corresponding  term  in  the  referent  map.  Currently,  the  synonyms  are  stored  in 
a  database  table  and  manually  specified  depending  on  the  context  of  the  particular 
task.  We  are  currently  integrating  WordNet  into  the  synonym  scoring  algorithm,  as 
the  WordNet  tables  contain  synonyms  for  over  100,000  words. 

Scoring  of  shot  groups.  The  second  automated  scoring  application  we  have 
developed  within  the  HPKMT  shell  is  to  score  shot-group  patterns.  As  shown  in 
Figure  2,  trainees  are  asked  to  depict  the  prototypical  shot  groups  of  five  types  of 
shooter  errors.  The  scoring  algorithm  compares  the  trainee's  response  against  an 
expert's  response  and  was  based  on  work  by  Johnson  (2001).  The  particular 
variables  compared  differ  by  shot  group,  but  in  general  include  dispersion  area, 
dispersion  radius,  vertical  displacement,  horizontal  displacement,  and  target 
quadrant.  A  complete  description  of  the  scoring  method  is  contained  in  Appendix 
D. 

Summary.  Our  original  design  approach  has  already  proved  useful.  The 
architecture  of  the  system  allowed  us  to  deliver  different  assessment  formats  within 
the  same  delivery  shell  using  the  same  codebase.  Automated  scoring  has  been 
implemented  for  each  format.  The  use  of  the  same  shell  to  deliver  assessments 
minimizes  user-interface  complexities. 
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We  have  developed  a  novel  application  that  moves  the  complexity  of  software 
development  to  the  scoring  aspect.  For  each  class  of  representation  types,  software 
needs  to  be  developed  to  score  that  representation.  But  this  is  much  easier  to  do  than 
creating  stand-alone  software  for  each  type  of  assessment.  Thus,  combinations  of 
icons  and  backgrounds  can  be  used  to  maximize  the  reuse  and  representational 
generalizability  of  the  tool. 

Usability  Study 

Because  we  anticipated  a  wide  range  of  users  of  the  mapper  software,  a 
usability  study  was  conducted  to  (a)  evaluate  how  intuitive  the  operation  of  the 
knowledge  mapper  was,  (b)  evaluate  how  learnable  the  knowledge  mapper 
operations  were,  and  (c)  uncover  error  and  atypical  usage  patterns. 

In  general,  the  usability  study  found  that  users  of  varying  levels  of  computer 
experience  learned  how  to  use  the  knowledge  mapper  with  minimal  instruction.  The 
most  difficult  function  of  the  mapper  to  learn  was  linking  concepts,  with  the  most 
users  requiring  suggestions  or  explicit  instruction  on  how  to  link.  Since  participants 
were  asked  to  use  and  explore  the  mapper  with  no  instruction  outside  of  an 
introduction  to  knowledge  mapping  in  general,  the  findings  suggest  that  a  short 
training  illustrating  the  basic  functions  of  the  mapper  would  be  sufficient  for  users 
with  even  limited  computer  experience.  Once  users  learned  how  to  use  the  mapper, 
there  were  few  recurring  usability  issues.  The  usability  study  yielded 
recommendations  for  changes  to  the  mapper,  including  modifications  to  the  linking 
function,  and  renaming  menu  items  and  toolbar  button  labels  for  clarity.  A  full 
report  is  given  in  Appendix  E.  A  preliminary  user  guide  is  provided  in  Appendix  F. 

Fielded  use.  We  tested  the  knowledge  mapper  in  an  assessment  context  with 
50  Marines  at  a  training  base  in  North  Carolina.  Most  of  the  Marines  were  between 
18  and  20  years  old.  Our  informal  observations,  based  on  observing  Marines  using 
the  software  and  Marines'  questions  and  comments,  suggested  that  the  Marines  had 
little  trouble  using  the  knowledge  mapper.  The  only  notable  question  was  how  to 
delete  objects,  as  some  users  attempted  to  delete  objects  with  the  backspace  key 
instead  of  the  delete  key. 
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Prototype  Authoring  Functionality 

To  support  the  creation  and  maintenance  of  knowledge  mapping  tasks,  a 
prototype  authoring  system  was  created.  The  design  of  the  system  was  based  on  the 
work  of  Chung  et  al.  (2002).  The  near-term  goal  of  the  authoring  functionality  was  to 
support  research  activities.  A  long-term  goal  was  to  develop  an  interface  that  is 
suitable  for  use  by  a  wide  audience  (e.g.,  trainers,  trainees,  researchers,  course 
managers). 

The  authoring  process  is  illustrated  in  Figure  2.  The  metaphor  adopted  was 
that  of  a  course.  A  course  consists  of  activities  or  tasks  that  students  engage  in.  There 
can  be  one  or  more  tasks  belonging  to  a  course.  An  activity  is  a  knowledge  mapping 
task.  Each  mapping  task  has  properties  associated  with  it,  such  as  the  set  of  concepts 
and  links,  the  set  of  icons  used,  and  the  mode  of  operation  (e.g.,  select-only,  type-in, 
or  both).  Tasks  can  be  created  by  using  existing  concepts  and  links  or  by  creating 
new  ones  or  augmenting  an  existing  set.  Trainees'  names  must  be  registered  in  the 
system  before  mapping  can  be  accessed.  Trainees  are  assigned  courses  (and 
indirectly  tasks).  In  the  following  sections,  each  major  functional  area  is  discussed 
with  accompanying  screen  shots. 
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Figure  3.  Top-level  flow  for  creating  a  new  knowledge  mapping  task. 


Top-Level  Screen 

Users  can  access  all  top-level  functions  from  the  main  screen  shown  in  Figure  4. 
The  design  of  the  Web  favored  breadth  over  depth,  which  allows  users  to  reach  their 
destination  within  4  clicks  (Chung  et  al.,  2002). 
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Course  Creation 

A  critical  step  in  the  authoring  process  is  setting  up  a  "course."  A  course 
bundles  one  or  more  tasks  and  can  be  assigned  to  trainees.  All  tasks  that  are  part  of  a 
course  are  made  accessible  to  the  trainee.  An  example  of  the  course  listing  screen  is 
shown  in  Figure  5.  A  summary  listing  is  shown  in  the  left  figure  and  a  detailed 
screen  is  shown  in  the  right  figure. 


1  - "" . v»  . ,,  x-« .  •  1 

Home  Course Task 

—  Back  *  Genetics/Oenetics 

poncept  Mapper  , 

Home  Course  Listings  : 

•  Bicvcle  Pumn/Bicvdc  Pi 

pD  Name 

•  Human  Phvsiolosrv/Hnm 

1  Test  Course 

Add  Task 

•  Civil  Engineering/Civilj 
«  Environmental  science/E 

•  Decision  Making 

•  Rifle  Marksmanshin/I.cic 

•  Human  Body 

•  Recycling 

•  Eye 

•  England  School  Test/Eni 

♦  Rifle  Marksmanship/Mat 

•  Pit\'?i cs  -  kinematics/For 

2  Rifle  Marksmanship 

Add  Task 

•  Rifle  Marksmanshin/Rifl 

•  Test  Course'Eye 

•  Rifle  Marksmanship 

•  Test  Course/Recvcline 

•  Marksmanship  June 

•  Test  Course/Human  Boc 

•  Lejeune  1 

•  Rifle  Marksmanship/Der 
»  Test  Cotirse/Decision  M; 

•  Demo  Marksmanship 

Add  Task 

•  Hearine/IIearine 

3  Physics  -  kinematics 

•  Vision/Vision 

•  Forces  in  motion 

! 

A  England  School  Test 
•  England  Task  1 

Add  Task 

Figure  5.  Assigning  tasks  to  courses. 
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Task  Creation 

To  create  a  task,  users  click  on  the  "add  task"  link  shown  in  Figure  5.  The  task 
creation  screen  appears  as  shown  in  Figure  6.  The  task  screen  allows  users  to  label  a 
task;  select  the  type  of  knowledge  representation;  and  specify  whether  the  student 
will  be  able  to  select  terms  and  links,  type  terms  and  links,  or  both.  This  screen  also 
allows  the  specification  of  the  set  of  concepts  and  relations  to  be  used,  as  well  as  the 
set  of  icons  to  be  used  for  mapping  and  the  background. 


Figure  6.  Specifying  task  attributes. 


Specifying  Concept  and  Link  Sets 

Specifying  the  set  of  concepts  and  relations  is  done  via  a  simple  form,  as  shown 
in  Figure  7.  This  allows  the  creation  of  sets  of  concepts  and  relations  that  can  be 
reused  in  other  tasks.  For  example,  one  use  of  this  feature  is  to  define  a  generic  set  of 
links  and  add  domain-specific  links.  In  this  way,  link  sets  can  be  quickly  created 
with  a  known  set  of  links,  and  tailored  to  specific  content  areas. 
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Concept  Mapper 
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Figure  7.  Specifying  task  attributes. 


Specifying  Image  Sets 

Analogous  to  specifying  concept  and  link  sets,  users  can  also  specify  image  sets 
for  the  mapping  symbol  set  and  the  background  symbol  sets.  Examples  of  these 
features  are  shown  in  Figure  8,  where  the  figures  on  the  left  show  the  interface  for 
specifying  mapping  icon  sets,  and  the  figures  to  the  right  show  the  user  interface  for 
the  background  images. 


Figure  8.  Adding  images. 
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Summary 

The  authoring  prototype  has  been  developed  to  facilitate  the  research  use  of  the 
knowledge  mapper.  The  prototype  contains  the  minimal  functionality  required  to 
allow  users  to  create  mapping  tasks  and  assign  tasks  to  students  and  other  trainees. 

Discussion 

We  have  developed  a  software  application  that  can  administer  and  score 
knowledge  maps  and  other  formats.  Our  usability  testing  and  our  field  test  suggest 
that  we  have  developed  an  easy-to-use  application.  Our  architecture  was  designed 
to  allow  different  assessment  formats  by  allowing  end-users  to  upload  icons  and 
background  images.  In  our  review  of  existing  mapping  software,  we  found  only  one 
other  application  focused  on  assessment,  and  we  found  no  other  application  that  has 
multiple  assessment  capability.  Our  use  of  knowledge  mapping  as  an  assessment 
tool  with  automated  scoring,  and  our  general  capability  to  support  unforeseen 
assessment  formats  are  unique.  CRESST  research  appears  to  be  the  only  empirically 
supported  online  mapping  tool  in  existence. 

Limitations  of  This  Work  and  Next  Steps 

There  are  two  major  limitations  of  this  work.  First,  the  authoring  component 
needs  to  be  recast  to  make  it  more  user-friendly  and  intuitive.  Currently,  specialized 
knowledge  and  training  are  needed  before  tasks  can  be  created.  There  is  no 
capability  to  specify  tasks  in  real-time  (e.g.,  using  the  mapper  itself  as  the  authoring 
tool).  The  second  limitation  is  the  lack  of  reporting.  We  currently  have  no  unified 
method  of  reporting  student  performance.  Scores  can  be  reported  on  a  single-user 
basis,  but  there  is  no  capability  for  group-level  reporting. 

Our  efforts  at  designing  a  usable  knowledge  mapping  tool  appear  to  be  a 
successful  first  step.  The  software  is  reasonably  stable  and  we  have  tested  the 
architecture  by  implementing  two  different  assessment  formats  with  scoring 
capability.  Clear  next  steps  include  retooling  the  authoring  prototype  interface.  A 
second  line  of  activity  is  to  deploy  the  mapper  in  distributed  learning  contexts  in 
different  content  areas  to  expand  the  suite  of  assessments. 
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Appendix  A 

Novak  and  Gowin's  Scoring  Guide 
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Reprinted  from  page  107  of  Novak  and  Gowin  (1984). 

(*)  Score  all  relationships  that  are  valid  (form  valid  propositions). 
You  may  wish  to  subtract  wrong  or  ambiguous  concept  linkages, 
but  tliis  may  complicate  scoring  unnecessarily.  Valid  cross-link 
relationships  might  be  scored  also,  or  simply  weighted  into  role 
r.  3  below. 

(2)  Count  valid  levels  of  hierarchy  and  score  each  level  X  times  as 
I  much  as  each  relationship,  lire  value  for  X  is  arbitrary,  but  because 
l  maps  have  many  more  relationships  than  hierarchies,  and  because 

valid  hierarchies  signal  progressive  differentiation  and  integrative 
reconciliation  of  concept  meanings,  it  seems  reasonable  and  valid 
to  score  each  level  of  the  hierarchy  3  to  10  times  as  much  as  each 
relationship.  Scoring  levels  of  hierarchy  when  maps  are  not  sym¬ 
metrical  is  a  problem,  but  wc  suggest  that  the  number  of  valid 
hierarchies  in  the  most  branched  segment  of  the  map  be  counted. 
Words  that  have  simply  been  strung  together  without  clear  sub* 
i:  ordinate  conceptual  relationships,  however,  should  not  count  as 

!  levels  of  hierarchies  (sec  Figure  5.5). 

(3)  Cross  links  that  show  valid  relationships  between  two  distinct 
segments  of  the  concept  hierarchy  signal  possibly  important 
integrative  reconciliations,  and  may  therefore  be  better  indicators 
of  meaningful  learning  titan  arc  hierarchical  levels.  We  suggest 
that  each  valid  cross  link  receive  2  or  3  times  the  points  assigned 
to  each  hierarchical  level.  Since  it  is  possible  to  construct  some 
kind  of  cross  link  between  almost  any  two  concepts  on  a  map, 
you  must  use  judgment  to  decide  if  a  given  cross  link  represents 
substantial  integrative  reconciliation  of  two  sets  of  concepts.  As 
with  essay  exams,  students  may  want  to  defend  their  answers,  and 
you  can  expect  some  negotiation  with  them  on  this  aspect  of 
scoring.  If  you  view  concept  mapping  primarily  as  a  learning  fool, 
diese  negotiations  might  be  as  valuable  to  the  student  as  the  map 

I;';'.:  construction.  ./ ^ 

(4)  It  may  be  desirable  to  ask  for  specific  examples  in  some  cases  to 
be  sure  that  the  students  know  w  hat  kind  of  event  or  object  is 
designated  by  the  concept  label.  You  may  w-ant  to  score  valid 
examples  the  same  as  relationships,  or  perhaps  give  them  half  as 
many  points  because  they  arc  usually  easy  to  supply  and  less  in¬ 
dicative  of  meaningful  learning.  It  is  possible  for  students  to  rote 
learn  that  a  cat  is  an  example  of  the  concept  mammal  (or  that  ice 
is  an  example  of  a  solid)  without  blowing  the  regularity  the  con¬ 
cept  designates.  Of  course,  because  concept  links  can  be  learned 
by  rote  as  well,  not  all  relationships  will  signal  meaningful  learn- 
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Tabic  2.4.  Scoring  criteria  for  concept  maps 


1.  Propositions.  Is  the  meaning  relationship  between  two  concepts 
indicated  by  the  connecting  line  and  linking  word{$)?  Is  the 
relationship  valid?  For  each  meaningful,  valid  proposition  shown, 
score  1  point.  (See  scoring  model  below.) 

2.  Hierarchy.  Docs  the  map  show  hierarchy?  Is  each  subordinate 
concept  more  specific  and  less  genera!  than  the  concept  drawn  above  it 
(in  the  context  of  the  materia!  being  mapped)?  Score  5  points  for  each 
valid  level  of  the  hierarchy. 

3.  Cross  links.  Docs  the  map  show  meaningful  connections  between  one 
segment  of  the  concept  hierarchy  and  another  segment?  Is  the 
relationship  shown  significant  and  valid?  Score  10  points  for  each  cross 
link  that  is  both  valid  and  significant  and  2  points  for  each  cross  link 
that  is  valid  but  does  not  illustrate  a  synthesis  between  sets  of  related 
concepts  or  propositions.  Cross  links  can  indicate  creative  ability  and 
special  care  should  be  given  to  identifying  and  rewarding  its 
expression.  Unique  or  creative  cross  links  might  receive  special 
recognition,  or  extra  points. 

4.  Examples:  Specific  events  or  objects  that  are  valid  instances  of  those 
designated  by  the  concept  label  can  be  scored  1  point  each.  (These  are 
not  circled  because  they  arc  not  concepts.) 

5.  In  addition,  a  criterion  concept  map  may  be  constructed,  and  scored, 
for  the  material  to  be  mapped,  and  the  student  scores  divided  by  the 
criterion  map  score  to  give  a  percentage  for  comparison.  (Some 
students  may  do  better  than  the  criterion  and  receive  more  than  100% 
on  this  basis). 
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Table  2.4.  (amt.) 


Scoring  Model 


Hierarchy 


Level  1 


Level  2 


Level  3 


Level  4 

Scoring  for  this  model: 
Relationships  (if  valid) 
Hierarchy  (if  valid)  4x5 
Cross  links  (if  valid 
and  significant)  10x2 
Examples  (if  valid)  4x1 


example  example 

J  I 

object  object 


cross  link 


— i —  crass 
link  link 


1  l  / 

example  example/ 

’  »  /  / 


specific 

concept 


specific 

concept 


58  points  total 


specific 

concept 
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Appendix  B 

Annotated  List  of  Existing  Mapping  Software 
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IHMC  Concept  Map  Software 

PC  / Mac  /  Solaris  /  Linux 
http:  /  /  cmap.coginst.uwf.edu/ 

This  software  is  for  academic  purposes,  including  collaborative  learning,  curriculum 
planning,  and  knowledge  organizing.  Users  can  share  propositions  in  maps  and 
form  discussion  groups  about  maps.  Other  advanced  features  of  the  software 
include  the  ability  to  add  images,  movies,  and  hyperlinks  to  concepts. 
Unfortunately,  the  increased  functionality  of  the  software  hinders  its  usability; 
creating  a  new  map  is  not  intuitive  enough  to  be  done  without  following  the  step- 
by-step  instructions  provided  in  the  Help  window  that  opens  upon  start-up.  This 
software  has  no  automated  scoring  capability;  it  appears  to  be  mainly  an 
instructional,  not  assessment,  tool,  best  used  for  organizing  thoughts  or  sharing 
ideas  within  a  group. 

Inspiration 

PC/Mac 

http:/  /  www.inspiration.com/productinfo/index.cfm 

This  software  is  for  academic  purposes  and  helps  teachers  implement  visual 
learning  techniques.  Students  can  use  maps  to  process  and  organize  information. 
Automated  scoring  is  not  available,  although  the  website  acknowledges, 
"Misdirected  links  or  wrong  connections  alert  educators  to  what  students  do  not 
understand,  providing  an  accurate,  objective  way  to  evaluate  areas  in  which 
students  do  not  yet  grasp  concepts  fully." 

Mind  Matrix 

PC/Mac 

http:  /  /  www.mind-matrix.org/ index? 

Targeted  to  the  academic  community,  this  software  offers  the  ability  for  instructors 
or  experts  to  create  tasks  and  then  administer  them  to  students.  Registered  Mind 
Matrix  users  can  share  and  view  tasks  and  maps  created  by  others.  Reporting 
capabilities  include  statistical  reports,  proposition  reports,  and  individualized 
student  reports.  Feedback  can  be  provided  to  students  based  on  whether  their 
propositions  match  propositions  in  the  expert  map. 

LifeMap 

PC  on  the  Web /Mac 

http:/  /  www2.ucsc.edu/ mlrg/ mlrgtools.html 

This  software  appears  to  be  intended  for  an  academic  audience.  Features  include 
exporting  maps  to  JPEG  format,  generating  Proposition  Databases  that  can  be  used 
to  generate  maps  based  on  all  the  propositions  in  the  database,  supporting 
QuickTime  movies  in  maps,  and  editing  by  multiple  users  at  different  locations  on 
the  same  map.  Building  a  map  is  not  as  straightforward  as  most  other  mapping 
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programs  that  have  the  look  and  feel  of  graphics  programs,  because  LifeMap  relies 
heavily  on  dialog  boxes.  There  does  not  appear  to  be  automated  scoring  available. 

Thinking  Maps 

PC /Mac 

http:/  /  www.thinkingmaps.com/httmsoftware.php3 

This  software  is  for  educational  purposes.  Thinking  Maps  are  asserted  to  promote 
continuous  development  and  integrated  thinking.  A  training  workshop  on  how  to 
create  and  use  maps  is  required,  as  there  is  a  specific  method  for  teaching  and 
assessing  Thinking  Maps.  Because  of  the  required  training,  however,  teachers  can 
use  Thinking  Maps  for  assessments  because  both  teachers  and  students  will 
understand  the  common  visual  language  in  the  maps,  and  therefore  students  from 
year  to  year  as  well  as  within  a  class  can  be  compared  to  each  other.  It  is  not  clear  if 
assessment  includes  the  ability  to  automatically  score  maps. 

Concept  Draw  MindMap 

PC /Mac 

http:/  /www.conceptdraw.com/ en/ products/ mindmap/main.php 
The  software  isn't  geared  to  any  one  community,  but  instead  offers  different  uses  for 
various  populations.  It  is  recommended  for  teachers  for  "preparing  lesson  plans" 
and  for  students  for  "thinking  over  essays  and  projects."  It  appears  to  be  mainly  a 
project  planning  or  brainstorming  tool.  It  features  export  capabilities  to  present 
maps  in  Powerpoint  or  put  them  on  the  Web,  formatting  functions  for  text  as  well  as 
graphics,  and  a  library  of  objects  as  well  as  the  ability  to  create  your  own. 

Semantica 

Mac 

http:  /  /  www.semanticresearch.com  /  index.html 

This  software  was  unavailable  for  evaluation.  The  software  is  not  geared  to  any 
particular  community.  The  software  is  intended  to  be  used  to  represent  a  knowledge 
base  that  can  be  shared  with  others.  The  website  says,  "Semantica  software  is 
intended  to  facilitate  the  transfer  and  sharing  of  knowledge.  This  can  be  from  expert 
to  learner  or  student,  or  among  peers,  experts  and  decision  makers.  In  an  eLearning 
environment,  Semantica  can  be  the  basis  of  stand-alone  courses,  an  adjunct  to  a 
blended  learning  episode,  or  a  reference  for  existing  texts  and  in-class  assignments." 
While  Semantica  can  be  used  for  instruction,  for  assessment  it  does  not  appear  to  be 
any  better  than  most  other  mapping  programs  that  lack  an  automated  scoring 
component. 

Visual  Mind 
PC 

http:/  /  www.visual-mind.com/ 
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The  software  appears  to  be  mainly  an  organizing  or  planning  tool.  Academic  uses 
include  note-taking,  studying,  and  outlining,  such  as  a  course  syllabus.  While  not 
targeted  at  one  specific  group,  it  appears  to  have  greater  functionality  for  business 
applications,  as  opposed  to  instruction  or  assessment.  The  ability  to  label  relations 
between  concepts  was  not  found.  No  automated  assessment  capability  appears  to  be 
available. 

Smart  Ideas 
PC 

http:  /  /  www.smarttech.com/products/smartideas/ 

The  software  says  students  can  use  concept  maps  to  organize  ideas.  Hyperlinks  can 
be  added  to  concepts  to  link  to  files  or  Web  sites.  Maps  can  have  multiple  levels  (for 
example,  you  can  have  multiple  propositions  embedded  within  a  concept).  This 
software  has  high  usability  and  the  interface  is  not  cluttered  with  too  many  features. 
No  automated  scoring  capability  appears  to  be  included. 

Mind  Mapper 

PC 

www.mindmapper.com 

Instructional  uses  for  Mind  Mapper  include  organizing  notes  (such  as  for  a  lecture 
by  a  teacher  or  from  a  lecture  by  a  student),  brainstorming,  organizing  complex 
thoughts  and  ideas,  outlining  papers,  or  testing  comprehension  (although  no 
automated  scoring  component  appears  to  be  provided).  Functions  include  the  ability 
to  recognize  outlines  in  text  files  and  import  them  as  maps,  and  the  ability  to  link 
Powerpoint  and  Excel  files  to  maps.  Intuitiveness  of  the  interface  was  low;  the 
ability  to  label  links  was  not  found. 

Fuzzy  Thought  Amplifier 
PC 

http://www.fuzzysys.com/ ftaprod.html 

No  evaluation  version  of  this  software  was  available,  so  usability  cannot  be  gauged. 
This  software  does  not  appear  to  be  for  any  particular  group.  Explicit  uses  for  the 
program  include  documenting,  manipulating,  and  communicating  ideas,  as  well  as 
for  spreadsheets,  bar  charts,  and  history  plots.  Applications  include  college  plans, 
legal  strategy,  and  traffic  control  design.  Features  include  mathematical  functions 
like  arctan  and  sine.  The  functionality  of  this  program  is  a  little  different  than  most 
because  of  the  spreadsheet  capabilities,  but  for  instruction  purposes,  it  does  not 
appear  to  offer  more  than  a  standard  mapping  program. 

Axon  Idea  Processor 
PC 

http:/ /web.singnet.com.sg/ ~axon2000/ 
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This  software  is  explicitly  designed  for  "planners,  writers,  managers,  and  students." 
It  is  mainly  for  organizing  ideas.  Features  include  3-dimensionality;  export 
capability  to  HTML,  XML,  EFM,  and  RTF;  drawing  and  graphics  capabilities;  and 
support  of  Hypertext  and  Richtext.  While  students  can  use  this  program,  it  is  not 
explicitly  for  instruction  and  does  not  feature  automated  assessment  capabilities. 
Usability  is  at  a  medium  level. 

Knowledge  Manager 
PC 

www.knowledgemanager.us 

This  program  is  not  directed  at  any  particular  group  of  users,  although  for  teachers, 
the  website  states,  sharing  knowledge  bases  over  the  Internet  "multiplies  his 
presence  and  didactic  capacities  with  minimum  effort,  even  in  distance  learning." 
Other  uses  include  decision  support  and  brainstorming.  No  evaluation  version  of 
this  software  was  available,  but  the  website  says  the  software  is  easy  to  use.  With 
this  software  you  can  share  with  a  group,  annotate  concepts,  organize  maps  into 
levels  (submaps),  export  to  text  or  image  files,  search  semantically,  open  a  map  form 
the  Web,  export  to  HTML,  make  relations  across  maps,  associate  external  documents 
or  applications  to  concepts.  No  automated  scoring  capability  was  found. 

Visio 

PC 

http :  /  /  www.microsoft.com  /  office  /  visio  /  default  .asp 

This  program  is  for  visualizing  and  sharing  ideas.  Visio  is  a  straightforward 
mapping  program  that  is  easy  to  use  and  does  not  have  an  overabundance  of 
features  to  weigh  down  the  software.  Creating  a  map  is  easy  to  accomplish  from 
start-up.  It  does  not  feature  an  automated  scoring  capability. 

Decision  Explorer 
PC 

http:  /  /  www.banxia.com  /  demain.html 

Geared  towards  business  applications  although  many  academic  institutions  (mainly 
universities,  colleges,  and  graduate  schools)  are  listed  as  users,  the  website  for  this 
software  lists  as  uses,  "providing  help  with  structuring  messy  or  complex  data  for 
problem  solving,  assisting  the  interview  process  by  increasing  understanding  and 
generating  agendas,  and  managing  large  amounts  of  qualitative  data  from 
documents."  Features  of  the  software  include  multiple  views  of  a  map,  functions  to 
hide  or  show  certain  parts  of  the  map  (e.g.,  groups  of  closely  linked  concepts, 
concepts  that  most  influence  outcomes),  and  the  ability  to  reduce  maps  to  provide 
an  overview  map  or  a  map  with  no  links  "to  focus  on  an  issue."  Usability  of  this 
program  was  low,  and  it  seemed  overburdened  with  features. 


Mind  Manager 
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PC 

http:  /  /  www.mindjet.com/index.shtml 

This  software  clearly  markets  corporations;  case  studies  listed  on  the  web  sites  boast 
of  "an  amazing  business  advantage"  and  are  worded  in  terms  of  cost  savings, 
employee  productivity,  and  customer  satisfaction.  The  software  allows  for  real-time 
sharing  of  maps.  Some  additional  features  include  ability  to  create  Web  sites  or  site 
maps,  project  tracking,  and  integration  with  Microsoft  Word,  Project,  PowerPoint, 
and  Outlook.  Usability  was  at  a  medium  level,  but  too  many  features  seem 
irrelevant  for  instructional  or  assessment  purposes.  It  does  not  appear  to  have  an 
automated  scoring  function. 

VisiMap 

PC 

http://www.coco.co.uk/ 

This  software  is  designed  for  brainstorming,  problem  solving,  outlining,  planning, 
and  meeting  facilitation.  Functionality  includes  spellcheck,  import/ export  ability 
from  picture  files  (e.g.,  JPEG,  TIFF),  and  a  separate  viewer  application  that  allows 
others  who  do  not  have  the  software  to  view  maps.  This  software  was  difficult  to 
use  and  seemed  to  have  too  many  features  than  would  be  used  in  a  classroom. 

SmartDraw 

PC 

http :  /  /  www.smartdraw.com  /  specials  /  diagram  ,asp?id=2387 

This  software  offers  the  capability  to  create  professional-looking  graphics  with  an 

emphasis  on  diagrams  and  flow  charts,  including  but  not  limited  to  organizational 

charts,  floor  plans,  and  engineering  schematics.  The  program  offers  a  library  of 

drag-and-drop  graphics,  such  as  all  50  U.S.  states.  While  the  software  is  easy  to  use, 

it  seems  better  applied  to  creating  professional-looking  pictures  than  knowledge 

maps. 
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Appendix  C 

Outline  of  the  Human  Performance  Knowledge  Mapper  Functional  Requirements 
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1 .  Icon  Set 

1.1.  Drag  to  canvas 

1.2.  Icon  labeled  to  denote  meaning  of  symbol 

2.  Background  set 

2. 1 .  Drag  to  canvas 

2.2.  Background  labeled  to  denote  meaning  of  background 

3.  Icon  operations 

3.1.  l=click  selects  object 

3.1.1.  Apply  weight 

3.1.2.  5-levels 

3.1.3.  Line  width  proportional  to  weight 

3.1.4.  Colors 

3.2.  Double-click  =  right  click 

3.2.1.  Popup  menu  of  concepts 
3 .2. 1 . 1  .Grayed  =  used 

3.2.1.2.Not  grayed  =  available 

3.2.2.  Text-box  fill-in 

3.2.3.  Use  increases  size  of  box  for  oversized  text 

3.3.  Drag  from  icon  panel  and  plop  on  canvas 

4.  Link  operations 

4.1.  l=click  selects  object 

4.1.1.  Apply  weight 

4.1.2.  5-levels 

4.1.3.  Line  width  proportional  to  weight 

4.1.4.  Colors 

4.2.  Link  label  operation 

4.2.1.  Weight 

4.2.2.  Label  selection  after  connection  in  dialog  box 

4.2.2. 1.  Cancel  =  no  link 

4.2.3.  Double-click  =  right  click 

4.2. 3.1.  Popup  menu  of  links 

4.2.3.2. Text-box  fill-in 


4.2.4.  Direction  of  arrows  (reverse) 
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5.  Network  operations 

5.1.  Delete  node 

5.1.1.  Delete  attached  1  inks 

5.2.  Explicit  click  on  link  mode  button 

5.3.  Explicit  click  on  arrow-button  for  move  and  delete  operations 

5.4.  Operations  applied  to  selected  objects 

5.5.  Zoom 

5.5.1.  Scale/%  and  default 

5.6.  Cut,  copy,  paste 

5.7.  Ctrl-drag 

5.8.  Group  selected  object  operations 

5.8.1.  Link  -  apply  selected  or  typed  in  term  to  all  links 

5.8.2.  General  properties  -  apply  property  to  all  selected  operations 

6.  Icon/Background  Panel 

6.1.  Tab-option 

6.1.1.  1st  choice:  vertical  tabs 

6.1.2.  2  tabs  -  one  for  “Icon”  one  for  “Background” 

6.1.3.  Populate  icon  panel  w/set  of  icons  relevant  to  task 

6. 1 .4.  Populate  background  panel  w/set  of  icons  relevant  to  task 

7.  Canvas 

7.1.  Multiple  canvases 

7.2.  Tabs  on  bottom 

7.3.  Size  of  infinite  w/scroll  capability 

8.  File  Operations 

8.1.  Saving 

8.1.1.  Dirty-bit 

8.1.2.  Options 

8. 1.2.1. Assessment/formal  -  save  when  exit,  no  prompt 
8. 1.2. 2. Informal  -  check  dirty-bit,  prompt  to  save 

8.1.3.  Save 

8.2.  Opening 

8.2.1.  Task 

8.2.2.  Map  (concept,  procedural,  physical) 
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8.2.3.  Options 

8. 2. 3.1.  Panel  for  tasks 

8.2.3.2. Panel  for  maps,  content  dependent  on  task  items 
9.  Printing  Operations 
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Appendix  D 

Shot  Group  Analyses  Scoring  Methodology 
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DCMsg  =  ,J(xs  -xe)2  +(ys  -ye)2 

For  the  purpose  of  this  report,  a  shot  group  may  be  considered  functionally  accurate 
if  the  constant  error,  DCMsg  is  less  than  or  equal  to  6. 

Shooting  Error  % 

The  average  of  the  separate  straight  lines  between  each  student's  shot  and  the  CM  of 
the  expert  shots. 


DCMS  =  — 


"CM  expert 


)  "i"  iy i  y  CM  exp  eri  ) 


DCMs  is  a  useful  measure  of  marksmanship  accuracy  because  it  gives  information 
on  whether  each  shot  actually  comes  close  to  the  center  of  the  target. 

Precision,  The  mean  Radius:  (Fig.  1) 

n 


Refers  to  the  average  of  the  distances  between  the  CSG  and  each  shot.  MR  gives  the 
shooter  important  information  about  the  overall  shot  group  tightness.  Shot  group 
tightness  is  a  popular  term  for  marksmanship  precision  and  is  the  magnitude  of 
dispersion  of  a  group  of  shots. 

In  scoring,  we  have  to  compare: 

MRstudent  ^  MRexpert 


the  max  permissible  value  for  |  MRstudent  -  MRexpert  |  ^  6. 

Area  of  Dispersion  (AD)  and  Diagonal  of  Dispersion  (DD)  (see  Figure  9) 


AD  (  X  Max  XMin)(y  Max  }’ Min) 


DD=  V(XMax  -XM,„)2  +(yMax  -yMin  f 
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For  each  expert  and  student  these  parameters  are  calculated  and  compared. 

ADstudent  ^  ADexpert 
DDstudent  ^  DDexpert 

Standard  deviation: 

a)  Horizontal  Component,  Sh 


Sh  = 


E 

xi~xl 

n- 1 

b)  Vertical  Component,  Sv 


Sv  = 


V 


n- 1 


c)  Radial  Component,  Sr 

Sr=  V(Sh)2+(Sv)2 

Standard  deviation  is  a  very  efficient  measure  of  variable  error. 

Scoring  Criteria  for  Shot-Group  Task 

Breathing  problem  (does  not  fire  during  the  natural  respiratory  pause). 

The  following  conditions  must  be  satisfied: 

1-  Shots  should  be  vertically  lined  up  around  the  center  line  x  =  0  with  (±1  unit) 
away. 

AD 

2-  ADstudent  vs  ADexpert  should  be  comparable  (  1 - mknL  <  25) 

ADexp  ert 

3-  Highs  AND  lows  of  the  student  and  expert  shots  should  not  be  more  than  1 
unit. 
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Bucking  the  rifle: 

1-  All  shots  must  be  located  at  the  third  quarter. 

2-  Shots  must  be  close  together  so  that  ADstudent  vs  ADeXpert  should  be 

AD 

comparable  (  1 - hihEhl  <  25) 

AD„V„„„ 


Flinching: 

1-  All  shots  must  be  located  at  the  first  quarter. 


Not  focusing  on  front  sight  tip: 


1-  All  shots  must  be  well  distributed  around  the  center  of  the  target  (0,0) 

2-  MRstudent  vs  MRexpcrt  must  be  comparable. 

3-  Shots  must  be  close  together  so  that  ADstudent  vs  ADexpert  should  be 


comparable  ( 


1- 


ADstit,/cn, 

perl 


<  25) 


Sight  adjustment: 

1-  With  the  wind  blowing  from  the  right,  all  the  shots  must  appear  in  the  far  left 

from  the  center,  around  the  line  y  =  0. 

2-  The  shots  must  be  as  close  as  possible  so  the  ADstudent  vs  ADexpert  should  be 

AD 

comparable  (  1 - <  25). 

F  AD  _ 
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Data  File  Format: _ 

Column  Description 


B 

Xs=Xmax  Xmax  of  the  shooter  shots 

C 

Xe=Xmax  Xmax  of  the  expert  shots 

D 

ratio  of  column  B  /  column  C 

E 

Xs=Xmin  Xmin  of  the  shooter  shots 

F 

Xe=Xmax  Xmin  of  the  expert  shots 

G 

ratio  of  column  E  /  column  F 

H-M  repeats  the 
same  for  the  Y 

N 

RHs  refers  to  the  Horizontal  range  for  the  shooter 

O 

RHe  refers  to  the  Horizontal  range  for  the  expert 

P 

ratio  of  Column  N/Column  O 

Q-S  repeats  the 
same  for  the 

vertical  ranges 

T 

ADs  is  the  area  covered  by  the  shooter  shots 

U 

ADe  is  the  area  covered  by  the  expert  shots 

V 

ratio  of  Column  T /Column  U 

w 

Xbar-s  is  the  X  coordinate  of  the  center  of  mass  for  the  shooter  shots 

X 

Xbar-e  is  the  X  coordinate  of  the  center  of  mass  for  the  expert  shots 

Y 

ratio  of  Column  W/ Column  X 

Z-AB  repeats  the 
same  for  the  Y 

coordinates 

AC 

MRs  is  the  mean  Radius  of  the  shooter  shots 

AD 

MRe  is  the  mean  Radius  of  the  expert  shots 

AE 

ratio  of  Column  AC/ Column  AD 

AF 

dsFromCenter  refers  to  the  distance  from  the  center  of  mass  to  point(0,0)  of 
the  shooter  shots 

AG 

deFromCenter  refers  to  the  distance  from  the  center  of  mass  to  point(0,0)  of 
the  expert  shots 

AH 

ratio  of  column  AF  /  column  AG 

AI 

alphaS  is  the  angle  between  horizontal  line  (where  y=0  and  x>0)  and  the  line 
from  the  center  of  mass  and  (0,0)  for  the  shooter. 

AJ 

alphaE  is  the  angle  between  horizontal  line  (where  y=0  and  x>0)  and  the  line 
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from  the  center  of  mass  and  (0,0)  for  the  expert. 

AK  ratio  of  Column  Al/Column  AJ 

AL  StandardDev_Horizontal 

AM  StandardDevJVertical 

AN  StandardDev_Radial 

AO  Scores 

AP  Details 

Note. 

1.  s  means  the  shooter  or  the  student,  and  e  means  the  expert. 

2.  With  the  pair  (dsFromCenter,  alphaS)  you  can  find  the  exact  orientation  of  the  center  of  mass  with 
respect  to  where  the  quarter  is  located.  This  information  is  particularly  important  for  scoring  Sight 
Adjustment  and  Flinching. 
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Appendix  E 

Knowledge  Mapper  Usability  Study 
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The  Knowledge  Mapper  created  by  the  National  Center  for  Research  on 
Evaluation,  Standards,  and  Student  Testing  (CRESST)  is  a  software  application  to 
create  knowledge  maps.  A  usability  study  was  conducted  to  measure  the 
intuitiveness  of  the  program,  as  well  as  its  usability  after  basic  functions  were 
learned. 

The  Mapper  can  be  operated  solely  with  the  mouse,  although  some  keyboard 
commands  are  functional.  Concepts  are  added  to  the  map  by  clicking  the  Concept 
icon  in  the  window  on  the  left  side  of  the  screen  and  dragging  it  to  the  canvas  (see 
Figure  El).  When  a  concept  is  added  to  the  map,  a  pop-up  menu  of  available 
concepts  is  provided  for  the  user  to  label  the  concept.  Links  between  concepts  are 
made  by  clicking  the  Connect  icon  (the  button  at  the  far  right  of  the  toolbar)  and 
then  drawing  a  line  from  one  concept  to  another.  Similar  to  labeling  concepts,  links 
are  labeled  from  a  pop-up  menu  of  available  links  that  is  provided  as  soon  as  a  link 
is  made.  Concepts  and  links  can  be  changed  by  either  double-clicking  or  right- 
clicking  on  the  concept  or  link  to  bring  up  the  label  menu. 


Figure  El.  Knowledge  Mapper  interface. 
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Concepts  and  links  can  be  deleted  by  clicking  on  Delete  Symbol  in  the  Edit 
menu  or  hitting  the  delete  key  on  the  keyboard.  Other  functions  in  the  Mapper 
include  line-formatting  capabilities,  such  as  changing  line  color  and  line  width,  and 
the  option  to  view  windows  that  show  concepts  and  links  available  for  the  task. 

Procedure 

Usability  testing  for  the  Knowledge  Mapper  was  conducted  from  Oct.  3  to  Oct. 
22,  2002,  with  14  participants.  Users  were  separated  into  three  categories  based  on 
results  of  the  computer  experience  measure  (Appendix  El):  High  users  had  extensive 
experience  with  graphics  programs  and  computers  in  general  (4  total),  medium  users 
had  minimal  exposure  to  graphics  programs  and  some  or  extensive  general 
computer  experience  (6  total),  and  loio  users  had  no  graphics  program  experience 
and  minimal  computer  experience,  limited  mainly  to  word  processing,  data  entry, 
and  Internet  access  (4  total). 

Participants  were  grouped  based  on  what  they  indicated  they  use  a  computer 
for,  as  well  as  the  icon  identification  portion  of  the  survey  on  the  last  two  pages. 
Icons  in  the  survey  represent  (a)  general  system  functions  that  appear  in  a  wide 
variety  of  software  applications  (e.g.,  undo,  copy),  (b)  drawing  or  formatting 
functions  that  appear  across  applications  (e.g.,  line  weight,  line  color),  and  (c)  higher 
level  graphics  functions  from  a  few  applications  or  sophisticated  functions  from 
graphics  programs  (e.g.,  group,  blur).  In  general,  participants  who  correctly 
identified  more  than  half  of  the  graphics  functions  were  classified  as  high  users, 
those  who  correctly  identified  some  of  the  functions  from  each  category  were 
classified  as  medium  users,  and  those  who  did  not  identify  any  of  the  graphics  and 
any  or  almost  any  of  the  drawing  functions  were  classified  as  low  users.  Because 
functions  in  the  Mapper  are  similar  to  those  in  drawing  and  graphics  applications, 
the  assumption  was  that  users  with  graphics  program  experience  would  learn  how 
to  use  the  Mapper  more  easily  than  those  users  who  had  no  exposure  to  such 
software. 

The  icon  identification  items  also  served  to  provide  additional  feedback  on  the 
recognizability  of  the  Knowledge  Mapper  icons,  as  all  of  the  toolbar  icons  featured 
in  the  Mapper  appeared  in  the  survey. 
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Participants  first  read  a  brief  introduction  to  knowledge  maps,  administered  on 
the  computer  with  PowerPoint  (see  Appendix  E2  for  knowledge  map  introduction). 
Then  users  were  administered  two  tasks  on  the  Knowledge  Mapper. 

For  Task  1,  users  were  given  no  instruction  on  the  Mapper  and  were  asked  to 
create  a  knowledge  map  on  the  human  body  (see  Appendix  E3  for  concepts  and 
links).  The  program  was  started  for  them  and  a  new  map  was  created  at  the  outset. 
They  were  given  a  maximum  of  20  minutes  for  this  task,  or  until  they  felt  they  could 
add  no  more  to  their  map.  Suggestions  or  instructions  were  provided  when  users 
either  exhibited  or  expressed  confusion. 

For  Task  2,  users  were  given  step-by-step  instructions  to  build  a  knowledge 
map.  Instructions  were  presented  in  a  separate  window  on  the  screen.  The  subject 
was  the  human  eye.  Instructions  were  intended  to  cover  most  of  the  functions  of  the 
Mapper  since  it  was  unlikely  users  would  discover  all  of  them  on  their  own  in  the 
first  task.  Instructions  were  phrased  as  actions  such  as  "Change  arrow  direction  of 
'controls'  so  the  relationship  reads  'brain  controls  eye"’ — users  were  not  explicitly  told  to 
find  the  arrow  direction  function  and  flip  the  arrowhead.  Because  of  the  wording  of 
the  instructions,  however,  users  often  found  workaround  solutions  to  some  actions; 
for  example,  with  the  above  instruction,  some  users  deleted  the  "controls"  link  and 
created  a  new  one  in  the  opposite  direction — the  instruction  was  followed,  although 
the  Arrow  Direction  function  was  not  sought  or  discovered.  This  means  that  some 
functions,  while  targeted  in  the  instructions,  were  not  discovered  or  used  by  all 
users.  For  this  task,  users  were  given  20  minutes  or  until  they  completed  the  last  step 
(17  steps  in  all;  see  Appendix  E4  for  task  steps). 

While  participants  completed  the  two  tasks,  an  observer  recorded  occurrences 
when  the  user  expressed  confusion  about  a  function  or  appeared  to  have  difficulty 
with  a  function.  After  completing  the  two  tasks,  participants  were  asked  questions 
about  the  functions  that  were  observed  as  difficult  to  use  or  understand.  After  the 
interview,  participants  completed  an  exit  survey  (Appendix  E5)  that  asked  more 
general  questions  about  the  Mapper  and  gave  an  additional  opportunity  for  open- 
ended  feedback. 

The  exit  survey  also  features  questions  about  the  eye-tracking  camera  that  was 
piloted  with  this  study.  The  eye  tracker  captured  the  participant's  gaze  and 
superimposed  it  on  a  videotape  that  recorded  the  participant's  computer  screen.  The 
data  from  the  eye  tracker,  however,  were  not  usable  because  (a)  the  eye  tracker 
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could  not  be  used  for  all  participants,  and  (b)  those  participants  whose  gaze  was 
captured  frequently  moved  out  of  range  of  the  camera  when  they  were  observed  to 
be  engaged  in  the  task. 


Results 

Table  El  presents  a  summary  of  errors  made  by  users  in  order  of  severity. 
Severity  is  determined  by  the  number  of  users  who  made  the  error  and  the  number 
of  times  the  error  occurred. 
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Table  El 

Knowledge  Mapper  Usability  Study  Error  Log 


No.  of  users  (no.  of  occurrences) 

High 

Medium 

Low 

Total 

(«  =  4) 

(n  =  6) 

(n  =  4) 

(N  =  14) 

Began  creating  link  or  tried  to  end  link  close  to  the  edge  of 
a  concept  but  without  completing  the  link 

1(1) 

5(17) 

4(14) 

10  (32) 

Tried  to  link  after  clicking  out  of  link  mode 

2(7) 

3(19) 

4(17) 

9(43) 

Clicked  on  concept  icon  to  add  concept  (more  than  one 
attempt) 

1(2) 

4(7) 

4(5) 

9(14) 

Linked  from  side  ports  of  a  concept  but  link  was  not 
completed 

2(5) 

2(21) 

4(5) 

8(31) 

Resized  concepts  instead  of  moving  or  linking  them 

1(3) 

2(12) 

4(6) 

7(21) 

Disconnected  link  from  concept  and  did  not  reconnect  it  to 
another  concept 

1(1) 

3(6) 

3(6) 

7(13) 

Created  a  link  without  labeling  it 

2(7) 

2(13) 

2(2) 

6(22) 

Clicked  on  link  direction  button  to  link 

1(6) 

3(15) 

1(1) 

5(22) 

Linked  concept  to  itself 

1(1) 

1(3) 

2(4) 

4(8) 

Clicked  on  items  in  concepts /links  list  to  add  a  concept  or 
link  to  the  map 

1(2) 

1(1) 

2(2) 

4(5) 

Dragged  concepts  under  left  pane  to  expunge  from  map 

0(0) 

1(2) 

2(7) 

3(9) 

Went  to  File-Delete  to  delete  a  concept/ link 

0(0) 

1(1) 

2(3) 

3(4) 

Resized  concept  to  thin  line  instead  of  moving  it 

0(0) 

1(1) 

2(2) 

3(3) 

Zoomed  out  at  start  of  task  so  when  concepts  were  dragged 
onto  the  canvas,  they  were  too  small  to  read 

0(0) 

1(1) 

2(2) 

3(3) 

Made  a  link  with  the  arrow  in  the  wrong  direction 

0(0) 

0(0) 

2(3) 

2(3) 

Clicked  on  Background  tab  and  didn't  click  back  to  Symbol 

0(0) 

1(1) 

1(1) 

2(2) 

Clicked  on  Select  button  to  link 

0(0) 

1(3) 

0(0) 

1(3) 

Moved  populated  concept  on  top  of  a  blank  concept  and 
made  links  to  the  blank  concept 

0(0) 

1(3) 

0(0) 

1(3) 

Selected  wrong  direction  to  switch  arrow  direction 

0(0) 

0(0) 

1(3) 

1(3) 

Made  links  bend  at  pivot  points  and  then  spent  time 
straightening  the  lines 

0(0) 

1(2) 

0(0) 

1(2) 

Attempted  to  make  a  link  with  only  one  concept  on  the 
map 

0(0) 

1(1) 

0(0) 

1(1) 

Dragged  link  labels  so  it  appeared  a  link  label  had  changed 

0(0) 

0(0) 

KD 

Id) 

Widened  left  pane  instead  of  moving  a  concept 

0(0) 

1(1) 

0(0) 

1(1) 
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In  short,  the  usability  study  found  that  users  of  varying  levels  of  computer 
experience  learned  how  to  use  the  Mapper  with  minimal  instruction.  The  most 
difficult  function  of  the  Mapper  to  learn  was  linking  concepts,  with  the  most  users 
requiring  suggestions  or  explicit  instruction  on  how  to  link.  Recommendations  for 
changes  to  the  Mapper  based  on  study  findings  include  modifications  to  the  linking 
function  to  head  off  some  of  the  problems  encountered  in  the  study,  in  addition  to 
other  areas  of  the  software,  such  as  suggestions  for  renaming  menu  items  and 
toolbar  button  labels  for  clarity. 

Since  participants  were  asked  to  use  and  explore  the  Mapper  with  no 
instruction  outside  of  an  introduction  to  knowledge  mapping  in  general,  the 
findings  suggest  that  a  short  training  illustrating  the  basic  functions  of  the  Mapper 
would  be  sufficient  for  users  with  even  limited  computer  experience.  Once  users 
learned  how  to  use  the  Mapper,  there  were  few  recurring  usability  issues. 

The  following  presents  in  more  detail  highlights  of  observations  of  testing 
sessions,  comments  from  participants,  and  recommendations  for  changes  to  the 
Mapper  as  well  as  suggestions  for  training  on  using  the  Mapper  and  creating 
knowledge  maps. 


Concepts 


Adding  Concepts 

Adding  concepts  to  the  map  was  observed  to  be  learned  after  trial  and  error. 
All  participants  clicked  once  on  the  Concept  icon,  without  dragging,  to  add  the 
concept  to  the  map.  Some  participants  tried  double-clicking  on  the  icon  to  add  it  to 
the  map.  Users  from  all  three  groups  (1  high,  4  medium,  4  low)  made  repeated 
attempts  to  single-  or  double-click  on  the  concept  icon  to  add  it  to  the  map  before 
abandoning  that  method.  Users  from  all  three  groups  were  given  suggestions  (1 
high,  1  medium,  3  low)  or  explicit  instructions  (1  medium,  1  low)  on  how  to  add 
concepts  to  the  map.  More  experienced  users  more  quickly  abandoned  unsuccessful 
efforts  to  add  concepts  and  dragged  the  icon  to  the  canvas  without  instruction. 

Users  across  all  three  groups  (1  high,  1  medium,  2  low)  went  to  View  => 
Concepts  and  tried  clicking  on  the  concept  names  to  add  them  to  the  map. 
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Two  users  (1  medium,  1  low)  clicked  on  the  Background  tab,  which  hid  the 
Concept  icon,  and  did  not  click  back  to  the  Symbol  tab  on  their  own. 

Users  from  all  three  groups  (1  high,  2  medium,  1  low)  suggested  the  instruction 
to  click  and  drag  the  concept  icon  should  be  made  explicit  through  training  or 
through  a  Help  feature. 

Once  users  learned  how  to  add  concepts,  no  further  instructions  or  reminders 
were  needed  as  all  users  completed  the  second  task  with  no  instruction  on  adding 
concepts. 

Recommendations .  Add  tooltip  "click  and  drag  to  map"  for  Concept  icon. 
Disable  ability  to  click  on  Background  tab  if  there  are  no  background  options 
provided. 

Labeling  Concepts 

All  users  successfully  labeled  concepts  with  no  instruction,  both  when  initially 
labeling  concepts  after  adding  them  to  the  map  and  also  when  renaming  concepts. 

Moving  Concepts 

Progress  on  creating  maps  was  hampered  when  users  clicked  on  corner  points 
of  concepts  and  resized  them  instead  of  moving  them  (1  high,  2  medium,  4  low). 
Time  was  spent  restoring  concepts  to  their  original  size  before  continuing  with  the 
task.  Three  users  (1  medium,  two  low)  resized  concepts  to  thin  lines  and  did  not 
expand  them,  blocking  their  ability  to  use  that  concept  in  their  map.  One  user  (low) 
dragged  a  resized  concept,  now  a  thin  vertical  line,  to  connect  two  other  concepts 
until  he  learned  how  to  use  the  Link  function. 

One  user  resized  concepts  for  the  Human  Body  map  to  create  a  representation 
of  the  body  on  the  canvas  (i.e.,  "body"  was  expanded  to  fill  most  of  the  screen  and 
organ  concepts  were  placed  within  the  borders  of  "body").  While  this  approach  to 
creating  a  knowledge  map  was  creative,  the  Mapper's  automated  scoring  function 
would  not  recognize  the  extra  effort  expended  on  the  pictorial  representation,  and 
therefore  time  and  thought  that  could  have  been  spent  refining  and  building  the 
knowledge  map  were  wasted. 

One  user  (medium)  moved  an  unlabeled  concept  on  top  of  a  labeled  concept 
twice.  Later  in  the  same  session,  she  moved  a  labeled  concept  on  top  of  the 
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unlabeled  concept.  The  unlabeled  concept  was  longer  than  the  labeled  concept,  so  it 
was  still  visible  from  underneath.  The  presence  of  unpopulated  concepts  was  not 
only  a  distraction,  but  would  have  posed  a  scoring  problem  when  the  user,  instead 
of  linking  to  the  labeled  concept  on  top,  created  links  that  adhered  to  the  blank 
concept  underneath. 

One  user  (high)  said  in  the  interview  that  it  should  not  be  possible  to  have 
unlabeled  concepts. 

Recommendations.  Limit  the  range  a  concept  can  be  resized.  If  a  concept  is  not 
labeled  after  it  is  dragged  to  the  map,  it  should  disappear. 

Links 


Adding  Links 

Adding  links  to  the  map  was  observed  to  be  more  challenging  than  adding 
concepts,  and  was  the  function  that  required  the  most  aid.  Users  from  all  three 
groups  (1  high,  5  medium,  3  low)  were  given  suggestions  or  explicitly  taught  to  link 
(1  high,  2  medium,  and  2  low  were  given  repeated  instruction  on  linking).  One  user 
(low)  who  successfully  linked  concepts  in  the  first  task  without  instruction  was 
taught  how  to  link  in  the  second  task  (he  said,  "I  didn't  know  what  I  was  doing). 
Another  user  (medium)  who  had  been  instructed  on  linking  during  the  first  task 
needed  a  reminder  for  the  second  task. 

In  the  post-task  interview,  4  users  (1  high,  3  medium)  cited  the  link  icon  as 
being  unfamiliar  and  therefore  linking  wasn't  very  intuitive  for  them.  Only  one  user 
(medium)  said  the  icon  is  appropriate  because  that  was  how  linking  should  be 
represented,  and  that  since  the  button's  tooltip  said  "connect,"  it  was  obvious  that 
was  the  button  to  use  for  linking.  This  user,  however,  also  needed  a  suggestion  to 
use  the  Link  button  before  he  began  linking  in  the  first  task. 

Four  users  (1  high,  3  medium,  1  low)  tried  to  use  the  Arrow  Direction  button  to 
create  links.  One  of  the  medium  users  made  repeated  efforts  across  tasks  to  use  the 
Arrow  Direction  button  and  was  the  same  user  as  above  who  was  given  a  reminder 
on  linking  in  the  second  task. 

One  user  (medium)  repeatedly  tried  to  link  using  the  Select  button,  which 
features  an  arrow  on  it. 
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Users  across  all  three  groups  tried  to  link  concepts  after  they  had  clicked  out  of 
the  link  mode  (2  high,  3  medium,  4  low),  most  of  them  repeatedly  either  within  the 
first  task  (2  high,  2  medium,  3  low)  or  across  both  tasks  (1  medium,  4  low).  One  user 
(high)  said  the  link  mode  should  be  locked  so  the  user  does  not  need  to  click  on  the 
button  for  each  link. 

Most  users  were  observed  to  link  concepts  from  edges,  as  opposed  to  the  center 
of  the  concept.  While  linking  was  for  the  most  part  successful,  two  issues  that 
hampered  progress  and  frustrated  users  arose. 

The  first  issue  involved  concepts  with  many  links  made  to  side  ports.  Users 
across  all  groups  (2  high,  2  medium,  4  low)  failed  to  complete  links  made  to 
concepts  with  many  links  (more  than  4  or  5  at  one  side),  even  though  they  initiated 
and  terminated  the  link  on  the  borders  of  the  concepts  and  had  successfully  linked 
earlier  using  this  method.  If  linking  from  the  center  port  was  not  discovered  on  their 
own,  when  instructed  to  connect  from  the  center  of  one  concept  to  the  center  of 
another,  users  linked  successfully,  but  most  first  repeated  linking  to  side  ports,  as 
that  was  the  learned  method  of  linking.  Impact  was  limited  to  those  users  with  more 
complex  maps  that  had  concepts  with  multiple  links,  but  because  of  the  number  of 
unsuccessful  attempts  made  by  users  to  complete  links  (high  users  made  4  and  1 
attempts  each,  medium  users  made  11  and  10  attempts  each,  1  low  user  made  2 
attempts,  and  3  low  users  made  1  attempt  each),  this  impeded  progress. 

The  second  issue  is  more  likely  to  be  fixed  by  training  and  instruction.  As  most 
users  used  side  ports  to  link,  some  users  (1  high,  5  medium,  4  low)  began  or  ended 
links  very  near  to  concept  borders  but  without  connecting  to  side  ports,  possibly  not 
noticing  the  spark  that  appears  on  the  concept  border  when  a  link  connection  has 
been  made.  Six  users  experienced  this  multiple  times  (4  medium,  2  low).  Instruction 
to  link  from  the  centers  of  concepts  would  likely  solve  this  problem  as  a  software  fix 
is  unfeasible. 

Four  users  (1  high,  1  medium,  2  low)  linked  a  concept  to  itself.  Because  one 
user  (low)  labeled  the  link,  the  relationship  appears  in  the  detail  of  the  map  (waste 
=>  carries  =>  waste).  The  user  had  at  first  related  two  concepts  with  the  link,  but 
then  dragged  the  origin  point  of  the  link  to  the  destination  concept,  linking  it  to 
itself.  Then  he  moved  the  link  label  so  it  was  in  between  the  concepts  he  had 
originally  linked,  giving  the  appearance  of  two  linked  concepts. 
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Recommendations.  The  tooltip  for  the  Link  button  should  be  changed  to  "link" 
to  aid  users  unfamiliar  with  the  icon  and  to  keep  language  consistent.  If  there  is  a 
limit  to  the  number  of  links  that  can  be  made  to  side  ports,  increase  the  limit. 
Training  should  emphasize  linking  from  the  centers  of  concepts.  Concepts  should  be 
restricted  from  being  linked  to  themselves. 

Labeling  Links 

Almost  all  users  labeled  links  successfully  on  their  own  (4  high,  5  medium,  4 
low).  Nevertheless,  6  users  (2  high,  2  medium,  2  low)  at  some  point  while  creating 
their  map  had  at  least  one  unlabeled  link. 

One  user  (medium)  created  links  and  repeatedly  clicked  out  of  the  pop-up 
menu  without  making  a  selection.  She  was  reminded  that  all  links  in  a  knowledge 
map  need  to  be  labeled. 

One  user  (high)  said  in  the  interview  that  it  should  not  be  possible  to  have 
unlabeled  links. 

Recommendation.  If  a  link  is  not  labeled  after  it  is  drawn,  it  should  disappear 
from  the  map. 

Changing  Link  Labels 

Two  steps  in  Task  2  asked  the  participant  to  change  a  link  label.  Four  users  (1 
high,  3  medium)  deleted  the  existing  link  and  created  a  new  one.  One  user  (low) 
dragged  the  text  label  from  a  nearby  link  to  another  so  it  appeared  on  the  screen  that 
the  link  had  changed,  when  in  reality  only  the  position  of  the  link  label  had  been 
moved. 

Recommendation .  Limit  the  range  a  link  label  can  be  moved. 

Moving  Links 

Some  users  moved  links  by  grabbing  end  points  and  dragging  them  to  other 
concepts,  reconnecting  them  successfully  most  of  the  time.  However,  7  users  (1  high, 
3  medium,  3  low)  disconnected  links  from  concepts  and  then  dragged  concepts  near 
or  on  top  of  the  end  point  of  the  link  to  simulate  a  complete  relationship. 

Recommendation.  If  a  link  is  disconnected  from  a  concept  and  not  reconnected 
to  a  concept,  it  should  disappear  from  the  map. 
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Changing  Link  Direction 

One  step  in  Task  2  asked  the  participant  to  change  the  arrow  direction  of  an 
existing  relationship.  Two  participants  (1  medium,  1  low)  skipped  this  step.  Four 
participants  (1  high,  3  medium)  deleted  the  existing  link  and  added  a  new  one  in  the 
opposite  direction.  One  participant  (medium)  dragged  the  end  points  of  the  link 
from  one  concept  to  the  other  to  reverse  the  arrow  direction.  Six  participants  (3  high, 
1  medium,  2  low)  successfully  changed  the  arrow  direction  on  their  own,  and  only 
one  user  (low)  needed  instruction  on  how  to  use  the  Arrow  Direction  button. 

One  user  (medium)  had  some  difficulty  using  the  Arrow  Direction  button  to 
reverse  the  arrow  direction  in  Task  2.  He  had  accidentally  reversed  the  arrow 
direction  of  the  wrong  relationship,  so  he  spent  some  time  trying  to  change  it  back. 
The  relationship  he  wanted  to  change  had  the  arrow  running  from  right  to  left.  He 
selected  the  link,  clicked  on  the  Arrow  Direction  button,  and  selected  the  first  arrow 
in  the  pop-up  menu,  as  it  runs  from  left  to  right  (the  new  direction  he  wanted  for  the 
link).  This  failed  to  change  the  direction  of  the  arrow.  He  repeated  the  actions 
multiple  times  without  success.  Moving  from  that  problem,  he  successfully  changed 
the  direction  of  the  arrow  in  the  task  instructions  as  that  arrow  ran  vertically  and  the 
first  option  he  chose  changed  the  direction. 

In  the  post-task  interviews,  two  users  (1  medium,  1  low)  said  changing  the 
arrow  direction  was  difficult. 

Recommendation.  Since  users  with  graphics  experience  had  no  trouble  with 
this  function,  it  seems  unwise  to  radically  change  the  function  of  the  button  from  the 
standard.  Only  one  user  (medium)  in  Task  1  (the  non-directed  task)  used  the  Link 
Direction  button.  Most  users  likely  will  never  need  to  use  this  function  or  will  be 
able  to  work  around  it  (deleting  and  replacing  links). 

Link  Pivot  Points 

One  user  (medium)  made  pivot  points  in  her  links  and  spent  time  straightening 
the  lines. 
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Concept  and  Link  Menus 
Access  to  Concept  and  Link  Menus 

Four  users  (2  medium,  2  low)  accessed  the  concept  and  link  menus,  but  they 
closed  them  after  attempting  to  add  concepts  using  them.  It  appeared  most  users 
added  new  concepts  or  links  or  clicked  on  existing  concepts  or  links  to  see  the 
options  available  for  building  maps. 

One  user  (medium)  said  viewing  relations,  but  not  necessarily  concepts,  should 
be  made  easier,  because  the  list  of  concepts  was  finite  and  after  they  were  all  added 
to  the  map,  you  would  not  need  to  see  a  list  of  concepts.  Because  relations  could  be 
reused,  however,  she  said  a  more  convenient  way  of  seeing  a  list  of  links  should  be 
provided,  since  creating  new  links  to  see  the  list  was  cumbersome.  When  informed 
the  View  menu  contained  the  lists  of  concepts  and  links,  she  said  that  was  not  where 
she  expected  to  find  them,  and  that  she  thought  if  a  list  of  links  was  built  into  the 
Mapper,  it  would  have  been  in  the  same  window  with  the  Concept  icon. 

One  user  (high)  said  she  expected  that  clicking  once  on  the  Concept  icon  would 
bring  up  a  list  of  available  concepts  and  she  would  have  preferred  that  option  rather 
than  having  to  drag  a  concept  to  the  map  and  commit  to  adding  a  new  concept. 

Recommendation.  Perhaps  buttons  to  view  concepts  and  view  relations  should 
be  added  to  the  toolbar. 


Edit  Menu  Functions 


Cut/Copy/Paste 

The  cut,  copy,  and  paste  functions  were  not  functional  on  the  version  of  the 
Mapper  in  the  usability  study.  Two  users  (one  high,  one  low)  commented  that  the 
Mapper  needs  these  functions. 

Delete 

The  Delete  function  was  the  second  most  difficult  function  on  the  Mapper  for 
users  to  perform,  with  more  than  half  of  the  users  needing  suggestions  on  how  to 
delete  (2  high,  2  medium,  4  low).  Users  were  directed  to  the  Edit  menu  if  they  were 
having  trouble  deleting;  most  found  the  "Delete  Symbol"  option  then.  One  user 
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(high)  was  directed  to  the  Edit  menu,  but  he  said  the  Delete  function  was  not  there; 
he  was  later  taught  how  to  delete. 

Three  users  (1  medium,  2  low)  were  observed  to  drag  concepts  under  the  left 
pane  to  expunge  them  from  the  map  instead  of  deleting  them. 

Three  users  (1  medium,  2  low)  went  to  File  =>  Delete  to  delete  a  concept  or 
link.  This  is  covered  in  detail  in  File  Menu  Functions:  Delete  Map. 

One  user  (medium)  commented  that  the  Delete  function  should  be  made  more 
obvious. 

Recommendations .  Training  should  include  instructing  users  that  they  can  use 
the  Delete  button  on  the  keyboard.  Also,  on  Mac  keyboards,  the  Delete  button  is  in 
the  same  spot  as  the  Backspace  key  on  a  PC  keyboard,  so  if  possible,  both  the 
Backspace  and  Delete  keys  should  delete  concepts  and  links.  The  Delete  Symbol 
menu  option  could  be  changed  to  "Delete  Concept /Link"  so  it  stands  out  more  to 
anyone  skimming  the  menu,  or  possibly  simply  "Delete." 

Undo/Redo 

Most  users  (2  high,  5  medium,  2  low)  used  the  Undo  /Redo  functions  in  the 
Mapper,  with  most  using  the  button  in  the  toolbar,  but  some  going  to  the  Edit  menu 
to  undo  actions.  Three  users  (2  medium,  1  low)  were  given  suggestions  on  the  Undo 
function. 

The  step  in  the  directed  task  targeting  the  Undo  function  asked  users  to  restore 
a  link  deleted  in  the  previous  step.  Five  users  (2  high,  1  medium,  2  low)  simply 
recreated  the  link  instead  of  using  Undo. 

Recommendation.  Most  users  found  and  utilized  the  Undo /Redo  functions  in 
the  Mapper.  However,  the  computer  experience  survey  shows  that  only  3  users 
correctly  identified  the  Redo  icon  and  4  users  correctly  identified  the  Undo  icon. 
Possibly  these  two  icons  could  be  redesigned  for  greater  recognizability. 
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Zoom 


Zoom  in/Zoom  Out 

Most  users  (4  high,  5  medium,  3  low)  successfully  zoomed  into  and  out  of  the 
map  without  help.  Three  users  (1  medium,  2  low)  were  given  a  suggestion  to  use  the 
magnifying  glass  icon  to  zoom  in. 

Three  users  (1  medium,  2  low)  zoomed  out  of  the  map  at  the  start  of  the  first 
task,  so  when  they  started  to  add  concepts  to  the  map,  the  concepts  were  too  small 
to  read.  The  users  were  guided  to  zoom  in  to  100%. 

One  user  (high)  said  the  zoom  function  was  not  convenient  because  if  you 
select  an  area  and  then  hit  the  zoom  button,  it  should  focus  on  that  area. 

Formatting 


Line  Weight 

Most  users  (4  high,  6  medium,  2  low)  successfully  changed  the  line  weight  per 
Task  2's  instructions  ("Make  the  border  of  the  concept  'Brain'  thicker").  Only  one 
user  (low)  needed  a  suggestion  before  changing  the  line  width. 

One  user  (low)  skipped  the  step  entirely. 

Another  user  (low)  passed  the  mouse  over  the  icon  and  "Line  weight"  popped 
up  as  a  tooltip,  but  he  did  not  click  on  the  button.  Instead  he  interpreted  the 
instruction  as  making  the  concept  larger  and  resized  the  concept.  In  the  post-task 
interview,  when  asked  if  he  would  have  clicked  on  the  button  if  the  label  said  "line 
width,"  he  said  that  would  have  made  more  sense  to  him. 

Recommendation.  Since  most  of  the  users  from  all  groups  had  no  trouble  with 
this  function,  it  may  be  unwise  to  deviate  from  the  standard  and  change  the  tooltip 
label. 

Line  Color 

Most  users  (4  high,  5  medium,  3  low)  successfully  changed  the  line  color  of  a 
link.  Only  2  users  needed  either  a  suggestion  (1  low)  or  to  be  taught  how  to  use  the 
function  (1  medium). 
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The  icon  for  line  color  was  the  only  icon  on  the  computer  experience  survey 
that  no  one  got  correct.  Most  users  did  not  attempt  to  label  the  icon  at  all;  the  two 
who  did  try  chose  the  wrong  answer.  Conversely,  the  Microsoft  Word  Line  Color 
icon  had  13  of  14  participants  label  it  correctly. 

Recommendation.  Change  the  Line  Color  icon  so  it  is  more  recognizable. 

Line  Style 

Most  users  (4  high,  6  medium,  3  low)  successfully  changed  the  line  style  of  a 
link.  Only  one  user  (low)  skipped  this  step. 

Other  Formatting  Issues 

In  the  exit  survey  and  post-task  interviews,  users  brought  up  other  issues 
regarding  formatting.  One  (low)  asked  for  the  ability  to  change  fonts  and  arrowhead 
appearance.  Another  (high)  asked  for  the  option  to  change  the  fill  color  of  the 
concepts. 


File  Menu  Functions 


Save  Map 

All  users  saved  the  map  successfully  on  their  own. 

Delete  Map 

Two  users  (low)  went  to  File  =>  Delete  to  delete  a  concept  or  link.  One  user 
chose  the  function  and  cancelled  out  of  the  dialog  window.  He  then  found  in  the 
Edit  menu  the  option  to  delete  a  concept  or  link.  Later  in  the  same  session,  however, 
the  user  went  to  File  =>  Delete  and  clicked  OK  without  paying  attention  to  the 
warning  message  in  red,  and  he  deleted  a  map. 

The  second  user  went  to  File  =>  Delete  but  was  instructed  before  he  clicked 
anything  to  cancel  out  of  the  window  to  prevent  any  other  maps  from  being  deleted. 

Recommendations .  The  menu  option  for  deleting  a  map  should  read  "Delete 
Map."  Also,  there  should  not  be  a  default  map  highlighted  for  deletion — if  a  user 
wants  to  delete  a  map,  the  map  should  be  specifically  chosen. 
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Miscellaneous  Comments 
User  Interface:  Background  Grid 

Two  users  in  the  post-task  interview  had  negative  comments  about  the  preset 
background  grid  on  the  canvas.  One  (medium)  said  the  grid  was  harsh  on  the  eye. 
Another  (high)  said  the  grid  was  unnecessary. 

User  Interface:  Symbols/Background  Pane 

One  user  (medium)  widened  the  left  pane  of  the  Mapper  window  instead  of 
grabbing  a  concept.  In  addition,  users  (1  medium,  2  low)  dragged  concepts,  some 
labeled,  under  the  left  pane  instead  of  deleting  them. 

Recommendations.  The  left  pane  should  be  a  fixed  width  and  impervious  to 
modification  by  the  user.  In  addition,  it  should  demarcate  the  left  border  of  the  map, 
as  opposed  to  lying  on  top  of  the  map,  so  users  cannot  leave  concepts  underneath 
the  pane. 

General  User  Interface  Issues 

Two  users  (1  medium,  1  low)  said  having  graphic  icons  for  the  concepts  (e.g.,  a 
brain  icon  and  a  heart  icon)  instead  of  boxes  would  make  it  more  visually  interesting 
for  users. 

Increased  Functionality — Right-Click 

Two  users  (1  high,  1  low)  said  right-clicking  on  a  concept  or  a  link  should 
provide  a  more  extensive  menu  with  more  options,  including  cut,  copy,  paste, 
delete,  as  well  as  the  concept  or  link  labels.  The  low  user  said  a  hint  or  help  function 
that  could  be  accessed  by  the  right-click  would  also  be  helpful. 

Miscellaneous  User  Recommendations 

Users  offered  general  recommendations  in  their  exit  surveys  and  post-task 
interviews.  One  (medium)  suggested  making  the  Mapper  more  like  a  game  would 
make  it  more  engaging. 

One  user  (low)  said  the  software  needs  a  Help  feature. 
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Observed  Bugs 

1.  An  unlabeled  concept  was  deleted,  but  the  outline  remained  on  the  canvas. 
Later,  the  user  was  able  to  rename  the  concept  and  integrate  it  into  the  map,  but  it 
was  the  only  transparent  concept  on  the  map  (one  occurrence). 

2.  No  matter  where  you  clicked  the  mouse,  the  link  would  begin  in  the  upper 
left  corner  of  the  canvas  in  the  same  spot,  until  the  Mapper  was  restarted  (two 
occurrences). 

3.  Clicked  on  paste  and  "<DOCT"  popped  up  on  canvas  (one  occurrence). 

Positive  Feedback 

Many  users  expressed  positive  feedback  about  the  Mapper,  calling  it  easy  to 
use  (1  high,  2  medium,  1  low),  self-explanatory  (2  medium,  1  low),  and  interesting  (1 
low).  One  user  explicitly  said  she  liked  it.  Two  (low)  said  the  Mapper  could 
potentially  be  useful.  Another  user  (medium)  was  impressed  that  the  Mapper  was 
sophisticated  enough  that  keyboard  commands,  such  as  control-Z,  worked. 

Negative  Feedback 

Some  users  said  the  Mapper  was  confusing  at  first  (2  low)  and  boring  (1 
medium). 


Knowledge  Mapping  Training 

Three  users  (1  high,  1  medium,  1  low)  left  unlabeled  links  in  their  maps  upon 
completion  of  the  task.  At  least  one  user  (low)  had  an  arrow  pointing  in  the  wrong 
direction.  One  user  (medium)  tried  to  make  a  link  when  there  was  only  one  concept 
on  the  map.  Several  users  needed  to  be  reminded  that  the  second  part  of  making  a 
knowledge  map  is  creating  links  between  concepts,  as  they  had  dragged  concepts  to 
the  map  and  only  worked  on  arranging  them  on  the  map  either  in  groups  or 
hierarchically,  before  asking  what  they  were  to  do  next. 

Some  changes  to  the  Knowledge  Mapper  will  mean  some  of  the  above  will  no 
longer  apply  (i.e.,  unlabeled  links).  Nevertheless,  an  introduction  to  knowledge 
mapping  must  emphasize  the  creation  of  links  between  concepts  as  well  as  the 
importance  of  arrow  direction. 
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UCLA  CRESST  Knowledge  Mapper 
KMT  Usability  Study 
Computer  Experience  Survey 

Your  responses  will  be  kept  completely  confidential.  The  results  of  this  survey  will  be  reported  only  for  this 
study  as  a  whole.  Your  particular  responses  will  not  be  identifiable  in  any  way. 


GENERAL  INFORMATION 


1 .  How  many  hours  per  week  do  you  use  a  computer?  _ hours 

2.  How  long  have  you  known  how  to  use  a  computer?  _ years 

3.  Do  you  use  a  computer  at  work?  □  Yes  □  No 

4.  Do  you  use  a  computer  at  home?  □  Yes  □  No 

5.  What  do  you  use  a  computer  for?  (Check  all  that  apply.) 


□  Word  processing 

□  Playing  games 

□  Desktop  publishing 

□  Visiting  Web  sites 

□  Creating  music  CDs 

□  Other 

□  E-mail 

□  Editing  movies 

□  Programming 

□  Data  entry 

□  Data  analysis 

□  Working  with  photos  or  graphics 

□  Working  with  databases 

□  Visiting  Internet  chat  rooms 

□  Creating  Web  site(s) 

□  Working  with  charts  or  tables 

6.  Do  you  use:  □  Mac  □  PC 

7.  Which  of  the  following  have  you  used?  (Check  all  that  apply.) 

Apple 

PC 

Other 

□  OS  9  or  lower 

□  DOS 

□  Unix/Linux 

□  OSX 

□  Other 


□  Windows  (any  version)  □  Other . 

□  Other _ 


Below  are  icons  for  buttons  that  are  used  in  a  variety  of  software  applications.  First,  indicate  how  confident 
you  are  that  you  know  what  the  icon  does  by  circling  either  “Don’t  know”  or  one  of  the  numbers.  Second. 
indicate  the  function  of  the  icon  by  using  the  numbered  list  given  to  you  and  writing  the  number 
corresponding  to  the  function  of  the  icon  in  the  last  column.  If  you  do  not  recognize  the  icon,  circle  “Don’t 
know”  and  leave  the  last  column  blank. 


How  confident  are  you  that  you  know  what  this  icon  does? 

Number 

Don’t  know 
this  icon 

Not  very 
confident 

Somewhat 

confident 

Confident 

Very 

confident 

corresponding  to 
icon’s  function 

p 

Don’t  know 

1 

2 

3 

4 

U ' 

Don’t  know 

1 

2 

3 

4 

\pt  T 

Don’t  know 

1 

2 

3 

4 

Don’t  know 

1 

2 

3 

4 

ji 

Don’t  know 

1 

2 

3 

4 

Don’t  know 

1 

2 

3 

4 

Don’t  know 

1 

2 

3 

4 

Don’t  know 

1 

2 

3 

4 

a. 

Don’t  know 

1 

2 

3 

4 

1  1 

1  1 
t-4* 

Don’t  know 

1 

2 

3 

4 

Don’t  know 

1 

2 

3 

4 

p 

Don’t  know 

1 

2 

3 

4 

1(31 

Don’t  know 

1 

2 

3 

4 

At 

Don’t  know 

1 

2 

3 

4 

|s 

Don’t  know 

1 

2 

3 

4 

m 

Don’t  know 

1 

2 

3 

4 

m 

Don’t  know 

1 

2 

3 

4 

Don’t  know 

1 

2 

3 

P 

Don’t  know 

1 

2 

3 

E 

Don’t  know 

1 

2 

3 

p 

Don’t  know 

1 

2 

3 

How  confident  are  you  that  you  know  what  this  icon  does? 


Don’t  know  Not  very  Somewhat  Very 

this  icon  confident  confident  Confident  confident 


Number 

corresponding  to 
icon’s  function 


Appendix  E2 

Knowledge  Map  Introduction 

Usability  Study 

•  We  are  asking  you  to  help  us  determine  the  usability  of  the  Knowledge 

Mapper  because  it  is  new  software. 

•  We  want  to  know  how  we  can  fix  it  to  make  it  easy  to  learn  and  easy  to  use. 

What  Is  a  Knowledge  Map? 

•  A  knowledge  map  measures  what  a  person  knows  about  a  subject. 

•  In  a  knowledge  map,  the  person  relates  concepts  to  each  other. 

Creating  a  Knowledge  Map 

•  A  set  of  concepts  and  relations  is  given  to  a  person  to  construct  a  knowledge 

map. 

•  For  example,  for  a  recycling  knowledge  map,  some  concepts  are  environment, 

pollution,  and  recycling.  The  list  of  relations  includes  helps  and  hurts. 

Linking  Concepts  and  Relations 

•  The  person  then  has  to  use  the  relations  to  link  the  concepts  together. 

•  Here  the  person  has  linked  pollution  and  environment  together  with  hurts, 

and  recycling  and  environment  with  helps. 

Tip:  Arrow  Direction  for  Relations 

•  Arrows  on  the  relations  show  the  direction  of  the  relationship. 

•  In  the  previous  example,  the  arrow  is  pointing  from  pollution  towards 

environment  because  that  is  the  direction  the  thought  is  read. 


•  To  have  the  arrow  point  the  opposite  way  would  have  the  relationship  read 

environment  ->  hurts  ->  pollution,  which  is  incorrect. 

Tip:  Relating  Concepts 

•  Only  one  relation  can  be  used  to  link  two  concepts  going  in  one  direction. 

•  However,  two  concepts  can  be  linked  twice,  if  the  relations  go  in  opposite 

directions. 

Tip:  Grammar  Does  Not  Matter 

•  Focus  on  the  meaning  of  a  relationship  without  thinking  about  grammar. 

•  For  example,  oceans  ->  is  a  part  of  ->  environment  will  be  scored  as  correct. 

What  We  Are  Measuring 

•  Usability  of  the  software  to  create  knowledge  maps 

What  We  Aren't  Measuring 

•  This  study  is  not  concerned  with  measuring  how  much  you  know  about  the 

subject — the  maps  you  create  will  not  be  evaluated. 

•  However,  we  are  still  asking  you  to  create  the  best  map  you  can,  as  we  would 

like  to  simulate  actual  test-taking  conditions. 

Questions? 

•  If  you  have  any  questions  at  this  time,  please  ask  the  researcher  now. 
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Human  Body  Concepts  and  Links 


Concepts  for  Task  1 

Links  for  Task  1 

Body 

Carries 

Brain 

Controls 

Stomach 

Digests  (breaks  down) 

Intestines 

Goes  to 

Food 

Is  made  of 

Waste 

Maintains 

Nutrients 

Is  part  of 

Energy 

Pumps 

Blood 

Removes 

Heart 

Tissues 

Appendix  E4 

Task  2  Step-by-Step  Instructions 

Step  1.  Create  the  relationship  "Optic  nerve  — >  Hits  — >  Brain" 

Step  2..  Create  the  relationship  "Eye  — >  Controls  — >  Brain" 

Step  3.  Create  the  relationship  "Light  — >  Focuses  — >  Retina" 

Step  4.  Create  the  relationship  "Retina  — >  Is  connected  to  — >  Optic  Nerve" 

Step  5.  Create  the  relationship  "Brain  — >  Creates  — >  Image" 

Step  6.  Change  the  arrow  direction  of  "Eye  — >  Controls  — >  Brain"  so  it  reads 
"Brain  — >  Controls  — >  Eye" 

Step  7.  Change  "Light  — >  Focuses  — >  Retina"  to  "Light  — >  Hits  — >  Retina" 

Step  8.  Make  the  line  around  the  concept  "Brain"  thicker 

Step  9.  Delete  the  relation  "Hits"  in  "Optic  nerve  — >  Hits  — >  Brain" 

Step  10.  Restore  the  relation  "hits"  in  "Optic  nerve  — >  Hits  — >  Brain" 

Step  11.  Change  "Hits"  to  "Sends  messages  to"  in  "Optic  nerve  — >  Hits  — > 
Brain" 

Step  12.  Change  the  color  of  the  relation  "Controls"  to  blue 
Step  13.  Change  the  relation  "Controls"  to  a  dotted  line 
Step  14.  Zoom  in  to  "Light  — >  Hits  — >  Retina" 

Step  15.  Delete  the  concept  "Light" 

Step  16.  Zoom  out  to  see  the  whole  map  on  the  screen 
Step  17.  Save  the  map  as  "Eye  Map" 


Appendix  E5 
Exit  Survey 


UCLA  CRESST  Knowledge  Mapper  Usability  Study 

Exit  Survey 

Your  responses  will  be  kept  completely  confidential.  The  results  of  this  survey  will  be  reported  only  for 
this  study  as  a  whole.  Your  particular  responses  will  not  be  identifiable  in  any  way. 


1 .  Age:  _ years 

2.  Gender:  □  Male 

3.  Ethnicity: 

□  Biracial/multiethnic 

□  African  American 

□  Asian  American 

□  Latino/Hispanic 


□  Female 

□  Native  American 

□  White,  non-Hispanic 

□  Other  _ 


4.  What  are  your  overall  impressions  of  the  Knowledge  Mapper? 

Please  check  one  item  in  each  row. 


Not  pleasing 

□  □ 

□  □ 

□ 

□ 

□ 

Pleasing 

Complicated 

□  □ 

□  a 

□ 

□ 

a 

Simple 

Difficult  to  use 

□  □ 

□  □ 

a 

□ 

□ 

Easy  to  use 

Incomplete 

□  □ 

□  □ 

a 

□ 

□ 

Complete 

How  familiar  were  vou  with  the  computer  task  topics? 

Please  circle  one  number  in  each  row. 

Not 

familiar 

Somewhat 

unfamiliar 

Somewhat 

familiar 

Familiar 

Very  familiar 

Human  body 

1 

2 

3 

4 

5 

The  eye 

1 

2 

3 

4 

5 

How  easv  or  difficult  did  vou  find  the  computer  task  topics? 

Please  circle  one  number  in  each  row. 

Very 

difficult 

Somewhat 

difficult 

Somewhat 

easy 

Easy 

Very 

easy 

Human  body 

1 

2 

3 

4 

5 

7.  How  easy  or  difficult  was  it  to  keep  your  head  still  for  the  eye  tracking  system  and 
understand  the  material  at  the  same  time? 

Very  Somewhat  Somewhat  Very 

difficult _ difficult _ easy _ Easy _ easy 

1  2  3  4  5 


8.  What  effects,  if  any,  did  the  eye  tracking  system  have  on  you  during  the  computer  task? 
Please  list  below  or  check  “no  effects”  if  there  were  no  effects. 

□  No  effects 


9.  What  would  you  change  on  the  Knowledge  Mapper? 
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Appendix  F 

Preliminary  Quick  Start  User  Guide 


Knowledge  Mapper  Quick  Start 
UCLA  CSE/CRESST 
July  2002 

Technical  Contact:  Dave  Brill,  brill@cse.ucla.edu 


This  is  a  short  guide  to  the  basic  features  of  the  CRESST  KnowledgeMapper.  It 
shows  you  how  to  create,  modify,  and  save  a  graphical  representation  of  the 
concepts  and  relations  in  a  knowledge  domain.  This  document  uses  screen  captures 
and  color-coded  annotations,  so  it  is  best  to  read  it  from  a  monitor  or  print  it  on  a 
color  printer.  A  more  detailed  User's  Guide  will  be  available  in  the  near  future. 


I  ’H  CRESST /UCLA  Concept  Mapper  -  Microsoft  Internet  Explorer 


File  Edit  View  Favorites  Tools  Help 
v3  Back  7  >4  ^  ^Search 


Address  ].||)  http://tripoli.cse,uda.edu:8080/devl/user/inde>;,jsp 


Use  your  regular 
browser  and  enter  this 
URL  in  the  address  bar. 


ISSIMM 


Flo  Ed*  View  Favorites  *  Tools  Help 
4- Back  -  •  <3  £}  ll^Searrt)  yjFavortes  ^Medre  (J  cJ  CS  lj 

Address  j^j  http://tripoli  cse .u< la  edu 1 8080/dpv i /user/nde x . ftp 

Go  flic  *  I  - ~  Search  Web  ^Search  5ke  I  P*^*RW  QPeqe  Info  «  (tt]Up  *  ; 


“BPP 


Search  for  other  items- 
Files  or  Folders 
Computet? 

People 

OOOOB  Merer  oft  Corpor»tion.  «  fight  I 
nstrvid.  I*naL0LUi5 


Choose  a  category  lor  your  search 
G  Find  a  i/eb  page 
C  Fjnd  a  person's  address 
C  Find  a  business 
C  Previoys  searches 
C  Find  a  map 

Mote .. 

I  a  Wah  page  containing-  j 


CRESST 

ll  he  National  Canter  for  Research  on  Evaluation,  Standards,  and  Student  Testing 


1 


Concept  Mapper 


Enter  your  Email  and  password  to  sign  in 
Email  Address  :  [msry@engiand 


About  CRESST  -  UCLA  ♦  CKflct  of  Njvjl  Research 


i<|S»rtj'  J  |gcRESST/Ua«  Concept.. 


f  1  j  j$  Internet  : 


When  the  Concept  Mapper 
page  appears  in  the  browser, 
enter  your  login  information. 

In  this  case,  the  login  address  is 
mary@england  and  the 
password  is  maryl. 


Enter  your  Email  and  password  to  sign  in 
Email  Address  :  |m  ary©  eng  I  and  j 
Password :  h~i  i 


o#B¥ffi»giBBQiBiiai&Qfl'e  ■»  «a  »  oe  »,  r 


If  this  is  the  first  time  that  you  are  using  the  KnowledgeMapper,  you  will  need  to 
install  Java  Web  Start.  Clicking  on  the  Install  Web  Start  link  takes  you  to  the  Sun 
site  from  which  vou  can  download  this  software.  This  onlv  needs  to  be  done  once. 


Address  |^]  http://tripoli.cse. ucla.edu:8080/devl/user/launch.jsp 

Go  glc  *  j  (^Search  Web  Q^SearchSite  |  5?::?  ©Pagelnfc  »  f^~)Up  *  Highlight 

Search  x 

:  G|  New  Next  v  j .  Custot  y> 

Jponcent  Mannar  ■  Elisabeth , Marv  Sian  Out-  CRESST 

Choose  a  category  for  your  search: 

Find  a  Web  page 

0  Fjnd  a  person's  address 

G  Find  a  business  / 

G  Previous  searches  * 

G  Find  a  map 

Map  Names 

Install  Web  Start  England  Map  1 

S _ s 

Java  Web  Start  is  a  new  technology  that  lets  you  run  applications  in  your  browser 
and  automatically  gets  updates  whenever  they  are  available.  Select  the  English 
Full  Package  option  and  then  follow  the  download  instructions.  If  you  don't  have 
the  current  version  of  Java  on  your  machine,  this  could  take  several  minutes. 


Search 


jjjg  Internet 


7:01  PM 


Java(lM)  web  Start  Download  -  Microsoft  Internet  Explorer 


Fie  Ed*  View  Favorites  Tools  He*p 

4*»*k  ♦  #  |g  (3  ||j§i^idT  syrwMM  9**'  &  t 


;  a  @  <>  it 


Address  |^j  h»p://|av8.sun.com/p’oductsh8vawetKtart/downlood-v«rxkiws.html 

Go  $e  -  j  ^ 

Search  X 

New  <%■  .-.  *  '  Cwtci  * 

Choose  a  category  foi  your  search: 
f*  Find  0  Ttfeb  page 
'  Fjnd  a  person's  address 
y  Find  a  business 
C  Previoys  searches 
’  Find  a  map 

Mcxs-. 


Find  a  Web  page  containing' 


Brought  to  you  by 
MSN  Search 


Search  j 


Search  lor  other  items: 

Files  or.  Folder? 

Compute; 

Ecccl? 

SQOOO  Meroiott  Corpowioo.  /HI  fijM  j 
r»l»rv»d.  T«rmr  of  Ut« 


pT^ . 

Jafistortjl  j 


Jp. Search  Web  ^-Search  Sffe  QPe&Mc,  -  g^Up 


lorhiiologim. 

hmvr.Ionds 

Docuri'irntotion 

Spotlight 

Developer  Services 
lav  s  liiuoPrinfs 


|  Download 
i  News 
|  README 
;  Developer's  Section 
:  Architecture 
:  Demos 
Partners  & 

.  Applications 

£2  Printable  Page 


^DOWNLOAD  FOR  MICROSOFT  WINDOWS  95  /  98  /  NT/2000  vl  .0.1  J02 


Java™  Web  Start  software  gives  you  the  power  to  launch  full-featured  applications  with  a  single  click 
from  your  Web  browser.  Please  follow  the  three  simple  steps  to  download,  install  and  y§g  the 
software,  but  before  you  start,  we  recommend  that  you  bookmark  this  page  to  come  back  easily. 

Step  1:  Download 

If  you  don't  have  the  Java™  Runtime  Environment  installed,  or  are  not  sure  whether  you  do,  you 
should  install  the  full  Java  Web  Start  package  (includes  the  JRE). 

If  you  do  have  the  Java  Runtime  Environment  version  1.2.2  or  higher  installed,  you  only  need  the 
Java  Web  Start  Update. 

Select  yourjj 


r 

Full  Package  (US-English  only) 

iLlI  Package  (International) 

Update  Package 

File  size:  5.7MB 

F*  size:  8.6MB 

File  size:  720KB 

j;:  continue  j 

j^ontinue  j 

pcontmiis^j 

f 

_|[g)3«*^TM)WcbS 


Now  click  the  Launch  link  to  start  the  KnowledgeMapper.  Whenever  you  use  the 
Mapper  after  the  first  time,  you  can  go  immediately  to  this  step. 


|  Address  |^}  http://tripoli.cse. uc!a.edu:808G/devl/user/launch.jsp 

Google "|  ^.Search web  ^Search SKe  i  P**fi«*  Q Rage  Info  «•  gjup  -  ^Highlight 

llWch  ,  . .  :  .  X 

New  Lc-v.t  -  |  Cusfcof'* 

Concept  Mapper  Elizabeth .  Marv  Sion  Out  *-CRESSf 

F  ^  "  '  .v  .  ‘ 

Choose  a  category  for  your  search: 

Find  a  Web  page  ■ 

O  Find  a  person's  address 

C  Find  a  business 

O  Previous  searches 

C  Find  a  map 

Launch  1  Map  Barnes 

Install  Web  Start  i  England  Map  1 

You  can  create  a  new  map  by  selecting  the  New  item  in  the  File  menu.  Or  you  can 
press  the  New  button,  which  is  the  first  button  on  the  toolbar. 


IIjCRESST  Knowledge  Mapper 


Ctil+N 


New  Map 


England  Task  1 


Cancel 


Ctrt+O 

j^'  Close 

f  (0  Save 

Qrt+S 

Save  As... 

Ctrl+SNft+S 

Delete... 

I?  Rename... 

If  1- '  Page  Setup. 

.  Ctrl+Shift+Q 

|  &  Print  i;: 

Ctrl+Q 

Every  map  belongs  to  a  content  area 
called  a  Task.  Tasks  are  assigned 
when  your  account  is  set  up.  Select 
which  task  you  want  the  new  map  to 
be  placed  in. 


Maps  are  created  by  dragging  icons  from  the  palette  (on  the  left  side  of  the 
Mapper)  to  the  canvas.  When  the  mouse  button  is  released,  a  text  box  appears  in 
the  new  node,  and  you  can  enter  the  name  of  the  concept  that  the  node 
represents.  In  some  tasks,  these  names  are  selected  from  a  list  rather  than  entered 
free-form.  Also,  some  tasks  have  several  different  icons  on  the  palette. 


^CRESST  Knowledge  Mapper 


Ctrl+Shift+S 


H^CRESST  Knowledge  Mapper 


Ctrl+O 


Ctrh-S 


Rename... 

Page  Setup,.,  Ctrl+Shlft+Q 
Print...  Ctrl+Q 


Heart 


Maps  are  saved  by  selecting  the 
Save  As  menu  item  (or  Save  if  you 
are  changing  an  old  map).  Enter  a 
name  for  the  new  map.  Then 
choose  Exit  if  you  are  finished. 


Enter  name  for  map: 


Physiology! 


Cancel 


Ctrl+N 

Ctrl+O 


©Save  :  Ctrl+S 
Save  As...  Ctrl+SNft+S 

Delete... 

: :  Page  Setup, ..  Ctrl+Shift+Q 
SPrint..,  Ctrl+Q 


Ctrl+O 


Kliiii 


Cancel 


ClNew 


View 

Ci 


jopenMap  >!■ 


Select  task: 

i(;:  Select  map  to  open: 


England  Map  1 


Saved  maps  are  stored  in  a  database 
on  a  CRESST  server.  You  can 
retrieve  air  old  map  by  selecting  the 
Open  item  on  the  File  menu,  or  by 
pressing  the  second  button  on  the 
toolbar.  When  the  Open  dialog 
appears,  first  choose  a  task  from  the 
upper  box,  and  then  choose  one  of 
the  maps  belonging  to  that  task. 
When  the  map  is  displayed,  you  can 
alter  or  extend  it,  and  then  save  it 
again.  To  save  a  map  under  its 
original  name,  select  the  Save  item  or 
press  the  third  button  on  the  toolbar. 


Close 


Save 


Ctrl+5 


Ctrl+5hift+S 


Save  As... 


Delete.. 


Rename... 


Page  Setup...  Ctrl+Sbift+Q 
Print...  Qrl+Q 


Exit 


CRESST  Knowledge  Mapper 


To  give  a  saved  map  a  new  name,  select  the  Rename  option  in  the  File  menu. 
In  the  Rename  dialog,  select  the  task  and  map  that  you  want  to  rename,  and 
then  tvoe  in  the  new  name. 


In  a  knowledge  map,  nodes  are  labeled  with  concept  names  and  links  are  labeled 
with  relation  names.  To  change  a  concept  name,  either  right-click  or  double-click 
the  node,  and  when  a  text  box  appears,  enter  the  new  name.  (For  some  tasks,  a 
list  may  appear  rather  than  a  text  box.)  Similarly,  in  order  to  change  a  relation 
name,  right-click  or  double-click  a  link  and  enter  the  new  name.  In  the  following 
example,  a  link  label  is  changed  from  pumps  to  circulates. 


* 


To  delete  a  node  or  a  link,  select  it  by  left-clicking  and  then  press  your  keyboard's 
delete  button.  (You  can  also  use  the  Delete  Symbol  item  in  the  Edit  menu.)  When 
you  delete  a  node,  you  also  remove  all  links  that  are  connected  to  it.  When  you 
delete  a  link,  the  connected  nodes  are  not  removed,  but  they  may  be  left  isolated. 
Unless  these  nodes  are  reconnected  to  the  map,  they  won't  be  considered  when 
the  map  is  scored. 


Nodes  can  be  moved  by  simply  dragging  them  across  the  canvas.  When  a  node  is 
moved,  all  attached  links  follow  it  to  the  new  location.  A  link,  however,  cannot  be 
moved  independently  of  the  nodes  that  it  connects. 


Groups  of  nodes  (and  attached  links)  can  be  moved  as  a  single  unit.  First  draw  a 
marquee  around  the  nodes  you  want  to  move.  Do  this  by  holding  the  left  mouse  button 
down  while  you  drag  the  mouse  diagonally  across  the  canvas.  Then  select  any  node 
within  the  marquee  and  drag  it  to  its  new  location.  The  other  nodes  will  follow.  This 
same  techniaue  can  be  used  to  delete  erouos  of  nodes  and  arcs  in  a  sinele  operation. 


Link  labels  are  usually  drawn  over  the  links.  If  these  labels  are  hard  to  read,  they  can  be 
moved  a  small  distance  by  dragging  them  with  the  mouse. 


Use  the  width  menu  to  change  the  width  of  an  object.  First  select  a  node  or  link,  or 
drag  a  marquee  around  a  group  of  nodes  and  links.  Then  choose  the  desired  line 
width  from  the  droodown  width  menu. 


If  you  change  your  mind  about  a  modification  you've  made  to  a  map,  click  the 
Undo  button  and  the  previous  state  of  the  map  will  be  restored.  By  clicking  Undo 
repeatedly,  you  can  go  back  as  many  steps  as  you  want.  If  you  go  too  far,  click 
Redo  one  or  more  times  to  reinstate  changes  that  you  made. 


The  zoom  controls  allow  you  to  change  the  size  of  the  nodes  and  links  in  the  map. 
The  Plus  and  Minus  magnify  buttons  increase  or  decrease  the  current  scale  by  a 
factor  of  two  each  time  they  are  pressed.  The  Zoom  dropdown  allows  you  to  select 
one  of  several  fixed  magnification  levels,  or  to  type  in  any  scale  that  you  want. 


We  hope  you  find  that  the  CRESST  KnowledgeMapper  is  a  useful  addition  to  your 
evaluation  tools.  Please  contact  us  with  any  questions  or  suggestions. 


